Обсуждение: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Hello Hackers,

This propose a way to develop "Table-level" Transparent Data Encryption (TDE) and Key Management Service (KMS) support
in
PostgreSQL.


Issues on data encryption of PostgreSQL
==========
Currently, in PostgreSQL, data encryption can be using pgcrypto Tool.
However, it is inconvenient to use pgcrypto to encrypts data in some cases.

There are two significant inconveniences.

First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.
Second, we must modify application program code much if we want to do database migration to PostgreSQL from other
databasesthat is
 
using TDE.

To resolved these inconveniences, many users want to support TDE.
There have also been a few proposals, comments, and questions to support TDE in the PostgreSQL community.

However, currently PostgreSQL does not support TDE, so in development community, there are discussions whether it's
necessaryto
 
support TDE or not.

In these discussions, there were requirements necessary to support TDE in PostgreSQL.

1) The performance overhead of encryption and decryption database data must be minimized
2) Need to support WAL encryption.
3) Need to support Key Management Service.

Therefore, I'd like to propose the new design of TDE that deals with both above requirements.
Since this feature will become very large, I'd like to hear opinions from community before starting making the patch.

First, my proposal is table-level TDE which is that user can specify tables begin encrypted. 
Indexes, TOAST table and WAL associated with the table that enables TDE are also encrypted.

Moreover, I want to support encryption for large object as well.
But I haven't found a good way for it so far. So I'd like to remain it as future TODO.

My proposal has five characteristics features of "table-level TDE".

1) Buffer-level data encryption and decryption
2) Per-table encryption
3) 2-tier encryption key management
4) Working with external key management services(KMS)
5) WAL encryption

Here are more details for each items.


1. Buffer-level data encryption and decryption
==================
Transparent data encryption and decryption accompany by storage operation 
With ordinally way like using pgcrypto, the biggest problem with encrypted data is the performance overhead of
decryptingthe data
 
each time the run to queries.

My proposal is to encrypt and decrypt data when performing DISK I/O operation to minimize performance overhead.
Therefore, the data in the shared memory layer is unencrypted so that performance overhead can minimize.

With this design, data encryption/decryption implementations can be developed by modifying the codes of the storage and
buffer
manager modules, 
which are responsible for performing DISK I/O operation.


2. Per-table encryption
==================
User can enable TDE per table as they want.
I introduce new storage parameter "encryption_enabled" which enables TDE at table-level.

    // Generate  the encryption table
       CREATE TABLE foo WITH ( ENCRYPTION_ENABLED = ON );

    // Change to the non-encryption table
       ALTER TABLE foo SET ( ENCRYPTION_ENABLED = OFF );

This approach minimizes the overhead for tables that do not require encryption options.
For tables that enable TDE, the corresponding table key will be generated with random values, and it's stored into the
newsystem
 
catalog after being encrypted by the master key.

BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this proposal. 
I'd like to hear opinions by security engineer.


3. 2-tier encryption key management
==================
when it comes time to change cryptographic keys, there is a performance overhead to decryption and re-encryption to all
data.

To solve this problem we employee 2-tier encryption. 
2-tier encryption is All table keys can be stored in the database cluster after being encrypted by the master key, And
masterkeys
 
must be stored at external of PostgreSQL.

Therefore, without master key, it is impossible to decrypt the table key. Thus, It is impossible to decrypt the
databasedata.
 

When changing the key, it's not necessary to re-encrypt for all data.
We use the new master key only to decrypt and re-encrypt the table key, these operations for minimizing the performance
overhead.

For table keys, all TDE-enabled tables have different table keys. 
And for master key, all database have different master keys. Table keys are encrypted by the master key of its own
database.
For WAL encryption, we have another cryptographic key. WAL-key is also encrypted by a master key, but it is shared
acrossthe
 
database cluster.


4. Working with external key management services(KMS)
==================
A key management service is an integrated approach for generating, fetching and managing encryption keys for key
control.
 
They may cover all aspects of security from the secure generation of keys, secure storing keys, and secure fetching
keysup to
 
encryption key handling.
Also, various types of KMSs are provided by many companies, and users can choose them.

Therefore I would like to manage the master key using KMS.
Also, my proposal is to create callback APIs(generate_key, fetch_key, store_key) in the form of a plug-in so that users
canuse many
 
types of KMS as they want.

In KMIP protocol and most KMS manage keys by string IDs. We can get keys by key ID from KMS. 
So in my proposal, all master keys are distinguished by its ID, called "master key ID". 
The master key ID is made, for example, using the database oid and a sequence number, like <OID>_<SeqNo>. And they are
managedin
 
PostgreSQL.
    
When database startup, all master key ID is loaded to shared memory, and they are protected by LWLock.

When it comes time to rotate the master keys, run this query.

    ALTER SYSTEM ROTATION MASTER KEY;

In this query, the master key is rotated with the following step.
1. Generate new master key,
2. Change master key IDs and emit corresponding WAL
3. Re-encrypt all table keys on its database

Also during checkpoint, master key IDs on shared memory become a permanent condition.


5. WAL encryption
==================
If we encrypt all WAL records, performance overhead can be significant.
Therefore, this proposes a method to encrypt only WAL record excluding WAL header when writing WAL on the WAL buffer,
insteadof
 
encrypting a whole WAL record.
WAL encryption key is generated separately when the TDE-enabled table is created the first time. We use 2-tier
encryptionfor WAL
 
encryption as well.
So, when it comes time to rotate the WAL encryption key, run this query.

    ALTER SYSTEM ROTATION WAL KEY;

Next, I will explain how to encrypt WAL.

To do this operation, I add a flag to WAL header which indicates whether the subsequent WAL data is encrypted or not.

Then, when we write WAL for encryption table we write "encrypted" WAL on WAL buffer layer.

In recovery, we read WAL header and check the flag of encryption, and judges whether WAL must be decrypted.
In the case of PITR, we use WAL key ID in the backup file.

With this approach, the performance overhead of writing and reading the WAL for unencrypted tables would be almost the
sameas
 
before.


==================
I'd like to discuss the design before starting making any change of code.
After a more discussion I want to make a PoC.
Feedback and suggestion are very welcome.

Finally, thank you initial design input for Masahiko Sawada.

Thank you.

[1] What does TDE mean?
    > https://en.wikipedia.org/wiki/Transparent_Data_Encryption

[2] What does KMS mean?
    > https://en.wikipedia.org/wiki/Key_management#Key_Management_System

[3] What does CBC-Mode mean?
    > https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
    
[4] Recently discussed mail
    https://www.postgresql.org/message-id/CA%2BCSw_tb3bk5i7if6inZFc3yyf%2B9HEVNTy51QFBoeUk7UE_V%3Dw%40mail.gmail.com


Regards.
Moon.
----------------------------------------
Moon, Insung
NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
----------------------------------------

Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:

This patch seems to implement some of the features you propose, especially
encryption of buffers and WAL. I recommend you to check so that no effort is
duplicated:

> [4] Recently discussed mail
>     https://www.postgresql.org/message-id/CA%2BCSw_tb3bk5i7if6inZFc3yyf%2B9HEVNTy51QFBoeUk7UE_V%3Dw%40mail.gmail.com



--
Antonin Houska
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com

Hello Moon,

I promised to email links to the articles I mentioned during your talk
on the PGCon Unconference to this thread. Here they are:

* http://cryptowiki.net/index.php?title=Order-preserving_encryption
* https://en.wikipedia.org/wiki/Homomorphic_encryption

Also I realized that I was wrong regarding encryption of the indexes
since they will be encrypted on the block level the same way the heap
will be.

--
Best regards,
Aleksander Alekseev

On Fri, May 25, 2018 at 8:41 PM, Moon, Insung
<Moon_Insung_i3@lab.ntt.co.jp> wrote:
> Hello Hackers,
>
> This propose a way to develop "Table-level" Transparent Data Encryption (TDE) and Key Management Service (KMS)
supportin
 
> PostgreSQL.
>
>
> Issues on data encryption of PostgreSQL
> ==========
> Currently, in PostgreSQL, data encryption can be using pgcrypto Tool.
> However, it is inconvenient to use pgcrypto to encrypts data in some cases.
>
> There are two significant inconveniences.
>
> First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.
> Second, we must modify application program code much if we want to do database migration to PostgreSQL from other
databasesthat is
 
> using TDE.
>
> To resolved these inconveniences, many users want to support TDE.
> There have also been a few proposals, comments, and questions to support TDE in the PostgreSQL community.
>
> However, currently PostgreSQL does not support TDE, so in development community, there are discussions whether it's
necessaryto
 
> support TDE or not.
>
> In these discussions, there were requirements necessary to support TDE in PostgreSQL.
>
> 1) The performance overhead of encryption and decryption database data must be minimized
> 2) Need to support WAL encryption.
> 3) Need to support Key Management Service.
>
> Therefore, I'd like to propose the new design of TDE that deals with both above requirements.
> Since this feature will become very large, I'd like to hear opinions from community before starting making the
patch.
>
> First, my proposal is table-level TDE which is that user can specify tables begin encrypted.
> Indexes, TOAST table and WAL associated with the table that enables TDE are also encrypted.
>
> Moreover, I want to support encryption for large object as well.
> But I haven't found a good way for it so far. So I'd like to remain it as future TODO.
>
> My proposal has five characteristics features of "table-level TDE".
>
> 1) Buffer-level data encryption and decryption
> 2) Per-table encryption
> 3) 2-tier encryption key management
> 4) Working with external key management services(KMS)
> 5) WAL encryption
>
> Here are more details for each items.
>
>
> 1. Buffer-level data encryption and decryption
> ==================
> Transparent data encryption and decryption accompany by storage operation
> With ordinally way like using pgcrypto, the biggest problem with encrypted data is the performance overhead of
decryptingthe data
 
> each time the run to queries.
>
> My proposal is to encrypt and decrypt data when performing DISK I/O operation to minimize performance overhead.
> Therefore, the data in the shared memory layer is unencrypted so that performance overhead can minimize.
>
> With this design, data encryption/decryption implementations can be developed by modifying the codes of the storage
andbuffer
 
> manager modules,
> which are responsible for performing DISK I/O operation.
>
>
> 2. Per-table encryption
> ==================
> User can enable TDE per table as they want.
> I introduce new storage parameter "encryption_enabled" which enables TDE at table-level.
>
>     // Generate  the encryption table
>        CREATE TABLE foo WITH ( ENCRYPTION_ENABLED = ON );
>
>     // Change to the non-encryption table
>        ALTER TABLE foo SET ( ENCRYPTION_ENABLED = OFF );
>
> This approach minimizes the overhead for tables that do not require encryption options.
> For tables that enable TDE, the corresponding table key will be generated with random values, and it's stored into
thenew system
 
> catalog after being encrypted by the master key.
>
> BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this
proposal.
> I'd like to hear opinions by security engineer.
>
>
> 3. 2-tier encryption key management
> ==================
> when it comes time to change cryptographic keys, there is a performance overhead to decryption and re-encryption to
alldata.
 
>
> To solve this problem we employee 2-tier encryption.
> 2-tier encryption is All table keys can be stored in the database cluster after being encrypted by the master key,
Andmaster keys
 
> must be stored at external of PostgreSQL.
>
> Therefore, without master key, it is impossible to decrypt the table key. Thus, It is impossible to decrypt the
databasedata.
 
>
> When changing the key, it's not necessary to re-encrypt for all data.
> We use the new master key only to decrypt and re-encrypt the table key, these operations for minimizing the
performanceoverhead.
 
>
> For table keys, all TDE-enabled tables have different table keys.
> And for master key, all database have different master keys. Table keys are encrypted by the master key of its own
database.
> For WAL encryption, we have another cryptographic key. WAL-key is also encrypted by a master key, but it is shared
acrossthe
 
> database cluster.
>
>
> 4. Working with external key management services(KMS)
> ==================
> A key management service is an integrated approach for generating, fetching and managing encryption keys for key
control.
> They may cover all aspects of security from the secure generation of keys, secure storing keys, and secure fetching
keysup to
 
> encryption key handling.
> Also, various types of KMSs are provided by many companies, and users can choose them.
>
> Therefore I would like to manage the master key using KMS.
> Also, my proposal is to create callback APIs(generate_key, fetch_key, store_key) in the form of a plug-in so that
userscan use many
 
> types of KMS as they want.
>
> In KMIP protocol and most KMS manage keys by string IDs. We can get keys by key ID from KMS.
> So in my proposal, all master keys are distinguished by its ID, called "master key ID".
> The master key ID is made, for example, using the database oid and a sequence number, like <OID>_<SeqNo>. And they
aremanaged in
 
> PostgreSQL.
>
> When database startup, all master key ID is loaded to shared memory, and they are protected by LWLock.
>
> When it comes time to rotate the master keys, run this query.
>
>         ALTER SYSTEM ROTATION MASTER KEY;
>
> In this query, the master key is rotated with the following step.
> 1. Generate new master key,
> 2. Change master key IDs and emit corresponding WAL
> 3. Re-encrypt all table keys on its database
>
> Also during checkpoint, master key IDs on shared memory become a permanent condition.
>
>
> 5. WAL encryption
> ==================
> If we encrypt all WAL records, performance overhead can be significant.
> Therefore, this proposes a method to encrypt only WAL record excluding WAL header when writing WAL on the WAL buffer,
insteadof
 
> encrypting a whole WAL record.
> WAL encryption key is generated separately when the TDE-enabled table is created the first time. We use 2-tier
encryptionfor WAL
 
> encryption as well.
> So, when it comes time to rotate the WAL encryption key, run this query.
>
>         ALTER SYSTEM ROTATION WAL KEY;
>
> Next, I will explain how to encrypt WAL.
>
> To do this operation, I add a flag to WAL header which indicates whether the subsequent WAL data is encrypted or
not.
>
> Then, when we write WAL for encryption table we write "encrypted" WAL on WAL buffer layer.
>
> In recovery, we read WAL header and check the flag of encryption, and judges whether WAL must be decrypted.
> In the case of PITR, we use WAL key ID in the backup file.
>
> With this approach, the performance overhead of writing and reading the WAL for unencrypted tables would be almost
thesame as
 
> before.
>
>
> ==================
> I'd like to discuss the design before starting making any change of code.
> After a more discussion I want to make a PoC.
> Feedback and suggestion are very welcome.
>
> Finally, thank you initial design input for Masahiko Sawada.
>
> Thank you.
>
> [1] What does TDE mean?
>     > https://en.wikipedia.org/wiki/Transparent_Data_Encryption
>
> [2] What does KMS mean?
>     > https://en.wikipedia.org/wiki/Key_management#Key_Management_System
>
> [3] What does CBC-Mode mean?
>     > https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
>
> [4] Recently discussed mail
>     https://www.postgresql.org/message-id/CA%2BCSw_tb3bk5i7if6inZFc3yyf%2B9HEVNTy51QFBoeUk7UE_V%3Dw%40mail.gmail.com
>
>

As per discussion at PGCon unconference, I think that firstly we need
to discuss what threats we want to defend database data against. If
user wants to defend against a threat that is malicious user who
logged in OS or database steals an important data on datbase this
design TDE would not help. Because such user can steal the data by
getting a memory dump or by SQL. That is of course differs depending
on system requirements or security compliance but what threats do you
want to defend database data against? and why?

Also, if I understand correctly, at unconference session there also
were two suggestions about the design other than the suggestion by
Alexander: implementing TDE at column level using POLICY, and
implementing TDE at table-space level. The former was suggested by Joe
but I'm not sure the detail of that suggestion. I'd love to hear the
deal of that suggestion. The latter was suggested by Tsunakawa-san.
Have you considered that?

You mentioned that encryption of temporary data for query processing
and large objects are still under the consideration. But other than
them you should consider the temporary data generated by other
subsystems such as reorderbuffer and transition table as well.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 06/11/2018 11:22 AM, Masahiko Sawada wrote:
> On Fri, May 25, 2018 at 8:41 PM, Moon, Insung
> <Moon_Insung_i3@lab.ntt.co.jp> wrote:
>> Hello Hackers,
>>
>> This propose a way to develop "Table-level" Transparent Data 
>> Encryption (TDE) and Key Management Service (KMS) support in 
>> PostgreSQL.
>>
>> ...
> 
> As per discussion at PGCon unconference, I think that firstly we
> need to discuss what threats we want to defend database data against.
> If user wants to defend against a threat that is malicious user who 
> logged in OS or database steals an important data on datbase this 
> design TDE would not help. Because such user can steal the data by 
> getting a memory dump or by SQL. That is of course differs depending 
> on system requirements or security compliance but what threats do
> you want to defend database data against? and why?
> 

I do agree with this - a description of the threat model needs to be 
part of the design discussion, otherwise it's not possible to compare it 
to alternative solutions (e.g. full-disk encryption using LUKS or using 
existing privilege controls and/or RLS).

TDE was proposed/discussed repeatedly in the past, and every time it 
died exactly because it was not very clear which issue it was attempting 
to solve.

Let me share some of the issues mentioned as possibly addressed by TDE 
(I'm not entirely sure TDE actually solves them, I'm just saying those 
were mentioned in previous discussions):

1) enterprise requirement - Companies want in-database encryption, for 
various reasons (because "enterprise solution" or something).

2) like FDE, but OS/filesystem independent - Same config on any OS and 
filesystem, which may make maintenance easier.

3) does not require special OS/filesystem setup - Does not require help 
from system adminitrators, setup of LUKS devices or whatever.

4) all filesystem access (basebackups/rsync) is encrypted anyway

5) solves key management (the main challenge with pgcrypto)

6) allows encrypting only some of the data (tables, columns) to minimize 
performance impact

IMHO it makes sense to have TDE even if it provides the same "security" 
as disk-level encryption, assuming it's more convenient to setup/use 
from the database.

> Also, if I understand correctly, at unconference session there also 
> were two suggestions about the design other than the suggestion by 
> Alexander: implementing TDE at column level using POLICY, and 
> implementing TDE at table-space level. The former was suggested by
> Joe but I'm not sure the detail of that suggestion. I'd love to hear
> the deal of that suggestion. The latter was suggested by
> Tsunakawa-san. Have you considered that?
> 
> You mentioned that encryption of temporary data for query processing 
> and large objects are still under the consideration. But other than 
> them you should consider the temporary data generated by other 
> subsystems such as reorderbuffer and transition table as well.
> 

The severity of those limitations is likely related to the threat model. 
I don't think encrypting temporary data would be a big problem, assuming 
you know which key to use.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi,

On 05/25/2018 01:41 PM, Moon, Insung wrote:
> Hello Hackers,
> 
> ...
> 
> BTW, I want to support CBC mode encryption[3]. However, I'm not sure 
> how to use the IV in CBC mode for this proposal. I'd like to hear
> opinions by security engineer.
> 

I'm not a cryptographer either, but this is exactly where you need a 
prior discussion about the threat models - there are a couple of 
chaining modes, each with different weaknesses.

FWIW it may also matter if data_checksums are enabled, because that may 
prevent malleability attacks affecting of the modes. Assuming active 
attacker (with the ability to modify the data files) is part of the 
threat model, of course.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 06/11/2018 05:22 AM, Masahiko Sawada wrote:
> As per discussion at PGCon unconference, I think that firstly we need
> to discuss what threats we want to defend database data against.

Exactly. While certainly there is demand for encryption for the sake of
"checking a box", different designs will defend against different
threats, and we should be clear on which ones we are trying to protect
against for any particular design.

> Also, if I understand correctly, at unconference session there also
> were two suggestions about the design other than the suggestion by
> Alexander: implementing TDE at column level using POLICY, and
> implementing TDE at table-space level. The former was suggested by Joe
> but I'm not sure the detail of that suggestion. I'd love to hear the
> deal of that suggestion.

The idea has not been extensively fleshed out yet, but the thought was
that we create column level POLICY, which would transparently apply some
kind of transform on input and/or output. The transforms would
presumably be expressions, which in turn could use functions (extension
or builtin) to do their work. That would allow encryption/decryption,
DLP (data loss prevention) schemes (masking, redacting), etc. to be
applied based on the policies.

This, in and of itself, would not address key management. There is
probably a separate need for some kind of built in key management --
perhaps a flexible way to integrate with external systems such as Vault
for example, or maybe something self contained, or perhaps both. Or
maybe key management is really tied into the separately discussed effort
to create SQL VARIABLEs somehow.

In any case certainly a lot of room for discussion.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
> Let me share some of the issues mentioned as possibly addressed by TDE
> (I'm not entirely sure TDE actually solves them, I'm just saying those
> were mentioned in previous discussions):

FYI, our product provides TDE like Oracle and SQL Server, which enables encryption per tablespace.  Relations, WAL
recordsand temporary files related to encrypted tablespace are encrypted.
 

http://www.fujitsu.com/global/products/software/middleware/opensource/postgres/

(I wonder why the web site doesn't offer the online manual... I've recognized we need to fix this situation.  Anyway, I
guessthe downloadable trial version includes the manual.)
 



> 1) enterprise requirement - Companies want in-database encryption, for
> various reasons (because "enterprise solution" or something).

To assist compliance with PCI DSS, HIPAA, etc.

> 2) like FDE, but OS/filesystem independent - Same config on any OS and
> filesystem, which may make maintenance easier.
> 
> 3) does not require special OS/filesystem setup - Does not require help
> from system adminitrators, setup of LUKS devices or whatever.
> 
> 4) all filesystem access (basebackups/rsync) is encrypted anyway
> 
> 5) solves key management (the main challenge with pgcrypto)
> 
> 6) allows encrypting only some of the data (tables, columns) to minimize
> performance impact

All yes.


> IMHO it makes sense to have TDE even if it provides the same "security"
> as disk-level encryption, assuming it's more convenient to setup/use
> from the database.

Agreed.


Regards
Takayuki Tsunakawa

> From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
> On 05/25/2018 01:41 PM, Moon, Insung wrote:
> > BTW, I want to support CBC mode encryption[3]. However, I'm not sure
> > how to use the IV in CBC mode for this proposal. I'd like to hear
> > opinions by security engineer.
> >
> 
> I'm not a cryptographer either, but this is exactly where you need a
> prior discussion about the threat models - there are a couple of
> chaining modes, each with different weaknesses.
Our products uses XTS, which recent FDE software like BitLocker and TrueCrypt uses instead of CBC.

https://en.wikipedia.org/wiki/Disk_encryption_theory#XTS

"According to SP 800-38E, "In the absence of authentication or access control, XTS-AES provides more protection than
theother approved confidentiality-only modes against unauthorized manipulation of the encrypted data.""
 



> FWIW it may also matter if data_checksums are enabled, because that may
> prevent malleability attacks affecting of the modes. Assuming active
> attacker (with the ability to modify the data files) is part of the
> threat model, of course.

Encrypt the page after embedding its checksum value.  If a malicious attacker modifies a page on disk, then the
decryptedpage would be corrupt anyway, which can be detected by checksum.
 


Regards
Takayuki Tsunakawa

On Wed, Jun 13, 2018 at 10:03 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> On 06/11/2018 11:22 AM, Masahiko Sawada wrote:
>>
>> On Fri, May 25, 2018 at 8:41 PM, Moon, Insung
>> <Moon_Insung_i3@lab.ntt.co.jp> wrote:
>>>
>>> Hello Hackers,
>>>
>>> This propose a way to develop "Table-level" Transparent Data Encryption
>>> (TDE) and Key Management Service (KMS) support in PostgreSQL.
>>>
>>> ...
>>
>>
>> As per discussion at PGCon unconference, I think that firstly we
>> need to discuss what threats we want to defend database data against.
>> If user wants to defend against a threat that is malicious user who logged
>> in OS or database steals an important data on datbase this design TDE would
>> not help. Because such user can steal the data by getting a memory dump or
>> by SQL. That is of course differs depending on system requirements or
>> security compliance but what threats do
>> you want to defend database data against? and why?
>>
>
> I do agree with this - a description of the threat model needs to be part of
> the design discussion, otherwise it's not possible to compare it to
> alternative solutions (e.g. full-disk encryption using LUKS or using
> existing privilege controls and/or RLS).
>
> TDE was proposed/discussed repeatedly in the past, and every time it died
> exactly because it was not very clear which issue it was attempting to
> solve.
>
> Let me share some of the issues mentioned as possibly addressed by TDE (I'm
> not entirely sure TDE actually solves them, I'm just saying those were
> mentioned in previous discussions):

Thank you for sharing!

>
> 1) enterprise requirement - Companies want in-database encryption, for
> various reasons (because "enterprise solution" or something).

Yes, I'm often asked it by our customers especially for database
migration from DBMS that supports TDE in order to reduce costs of
migration.

>
> 2) like FDE, but OS/filesystem independent - Same config on any OS and
> filesystem, which may make maintenance easier.
>
> 3) does not require special OS/filesystem setup - Does not require help from
> system adminitrators, setup of LUKS devices or whatever.
>
> 4) all filesystem access (basebackups/rsync) is encrypted anyway
>
> 5) solves key management (the main challenge with pgcrypto)
>
> 6) allows encrypting only some of the data (tables, columns) to minimize
> performance impact
>
> IMHO it makes sense to have TDE even if it provides the same "security" as
> disk-level encryption, assuming it's more convenient to setup/use from the
> database.

Agreed.

>
>> Also, if I understand correctly, at unconference session there also were
>> two suggestions about the design other than the suggestion by Alexander:
>> implementing TDE at column level using POLICY, and implementing TDE at
>> table-space level. The former was suggested by
>> Joe but I'm not sure the detail of that suggestion. I'd love to hear
>> the deal of that suggestion. The latter was suggested by
>> Tsunakawa-san. Have you considered that?
>>
>> You mentioned that encryption of temporary data for query processing and
>> large objects are still under the consideration. But other than them you
>> should consider the temporary data generated by other subsystems such as
>> reorderbuffer and transition table as well.
>>
>
> The severity of those limitations is likely related to the threat model. I
> don't think encrypting temporary data would be a big problem, assuming you
> know which key to use.

Agreed. I thought the possibility of non-encrypted temporary data in
backups but since we don't include them in backups it would not be a
big problem.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Jun 13, 2018 at 10:20 PM, Joe Conway <mail@joeconway.com> wrote:
> On 06/11/2018 05:22 AM, Masahiko Sawada wrote:
>> As per discussion at PGCon unconference, I think that firstly we need
>> to discuss what threats we want to defend database data against.
>
> Exactly. While certainly there is demand for encryption for the sake of
> "checking a box", different designs will defend against different
> threats, and we should be clear on which ones we are trying to protect
> against for any particular design.
>
>> Also, if I understand correctly, at unconference session there also
>> were two suggestions about the design other than the suggestion by
>> Alexander: implementing TDE at column level using POLICY, and
>> implementing TDE at table-space level. The former was suggested by Joe
>> but I'm not sure the detail of that suggestion. I'd love to hear the
>> deal of that suggestion.
>
> The idea has not been extensively fleshed out yet, but the thought was
> that we create column level POLICY, which would transparently apply some
> kind of transform on input and/or output. The transforms would
> presumably be expressions, which in turn could use functions (extension
> or builtin) to do their work. That would allow encryption/decryption,
> DLP (data loss prevention) schemes (masking, redacting), etc. to be
> applied based on the policies.

It seems good idea. Which does this design encrypt data on, buffer or
both buffer and disk? And does this design (per-column encryption) aim
to satisfy something specific security compliance?

> This, in and of itself, would not address key management. There is
> probably a separate need for some kind of built in key management --
> perhaps a flexible way to integrate with external systems such as Vault
> for example, or maybe something self contained, or perhaps both.

I agree to have a flexible way in order to address different
requirements. I thought that having a GUC parameter to which we store
a shell command to get encryption key is enough but considering
integration with various key managements seamlessly I think that we
need to have APIs for key managements. (fetching key, storing key,
generating key etc)

> Or
> maybe key management is really tied into the separately discussed effort
> to create SQL VARIABLEs somehow.
>

Could you elaborate on how key management is tied into SQL VARIABLEs?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 06/14/2018 12:19 PM, Masahiko Sawada wrote:
> On Wed, Jun 13, 2018 at 10:20 PM, Joe Conway <mail@joeconway.com> wrote:
>> The idea has not been extensively fleshed out yet, but the thought was
>> that we create column level POLICY, which would transparently apply some
>> kind of transform on input and/or output. The transforms would
>> presumably be expressions, which in turn could use functions (extension
>> or builtin) to do their work. That would allow encryption/decryption,
>> DLP (data loss prevention) schemes (masking, redacting), etc. to be
>> applied based on the policies.
> 
> Which does this design encrypt data on, buffer or both buffer and
> disk?


The point of the design is simply to provide a mechanism for input and
output transformation, not to provide the transform function itself.

How you use that transformation would be entirely up to you, but if you
were providing an encryption transform on input the data would be
encrypted both buffer and disk.

> And does this design (per-column encryption) aim to satisfy something
> specific security compliance?


Again, entirely up to you and dependent on what type of transformation
you provide. If, for example you provided input encryption and output
decryption based on some in memory session variable key, that would be
essentially TDE and would satisfy several common sets of compliance
requirements.


>> This, in and of itself, would not address key management. There is
>> probably a separate need for some kind of built in key management --
>> perhaps a flexible way to integrate with external systems such as Vault
>> for example, or maybe something self contained, or perhaps both.
> 
> I agree to have a flexible way in order to address different
> requirements. I thought that having a GUC parameter to which we store
> a shell command to get encryption key is enough but considering
> integration with various key managements seamlessly I think that we
> need to have APIs for key managements. (fetching key, storing key,
> generating key etc)


I don't like the idea of yet another path for arbitrary shell code
execution. An API for extension code would be preferable.


>> Or
>> maybe key management is really tied into the separately discussed effort
>> to create SQL VARIABLEs somehow.
> 
> Could you elaborate on how key management is tied into SQL VARIABLEs?

Well, the key management probably is not, but the SQL VARIABLE might be
where the key is stored for use.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Wed, Jun 13, 2018 at 9:20 AM, Joe Conway <mail@joeconway.com> wrote:
>> Also, if I understand correctly, at unconference session there also
>> were two suggestions about the design other than the suggestion by
>> Alexander: implementing TDE at column level using POLICY, and
>> implementing TDE at table-space level. The former was suggested by Joe
>> but I'm not sure the detail of that suggestion. I'd love to hear the
>> deal of that suggestion.
>
> The idea has not been extensively fleshed out yet, but the thought was
> that we create column level POLICY, which would transparently apply some
> kind of transform on input and/or output. The transforms would
> presumably be expressions, which in turn could use functions (extension
> or builtin) to do their work. That would allow encryption/decryption,
> DLP (data loss prevention) schemes (masking, redacting), etc. to be
> applied based on the policies.

It seems to me that column-level encryption is a lot less secure than
block-level encryption.  I am supposing here that the attack vector is
stealing the disk.  If all you've got is a bunch of 8192-byte blocks,
it's unlikely you can infer much about the contents.  You know the
size of the relations and that's probably about it.  If you've got
individual values being encrypted, then there's more latitude to
figure stuff out.  You can infer something about the length of
particular values.  Perhaps you can find cases where the same
encrypted value appears multiple times.  If there's a btree index, you
know the ordering of the values under whatever ordering semantics
apply to that index.  It's unclear to me how useful such information
would be in practice or to what extent it might allow you to attack
the underlying cryptography, but it seems like there might be cases
where the information leakage is significant.  For example, suppose
you're trying to determine which partially-encrypted record is that of
Aaron Aardvark... or this guy:
https://en.wikipedia.org/wiki/Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff,_Sr.

Recently, it was suggested to me that a use case for column-level
encryption might be to prevent casual DBA snooping.  So, you'd want
the data to appear in pg_dump output encrypted, because the DBA might
otherwise look at it, but you wouldn't really be concerned about the
threat of the DBA loading a hostile C module that would steal user
keys and use them to decrypt all the data, because they don't care
that much and would be fired if they were caught doing it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 06/18/2018 09:49 AM, Robert Haas wrote:
> On Wed, Jun 13, 2018 at 9:20 AM, Joe Conway <mail@joeconway.com> wrote:
>>> Also, if I understand correctly, at unconference session there also
>>> were two suggestions about the design other than the suggestion by
>>> Alexander: implementing TDE at column level using POLICY, and
>>> implementing TDE at table-space level. The former was suggested by Joe
>>> but I'm not sure the detail of that suggestion. I'd love to hear the
>>> deal of that suggestion.
>>
>> The idea has not been extensively fleshed out yet, but the thought was
>> that we create column level POLICY, which would transparently apply some
>> kind of transform on input and/or output. The transforms would
>> presumably be expressions, which in turn could use functions (extension
>> or builtin) to do their work. That would allow encryption/decryption,
>> DLP (data loss prevention) schemes (masking, redacting), etc. to be
>> applied based on the policies.
> 
> It seems to me that column-level encryption is a lot less secure than
> block-level encryption.  I am supposing here that the attack vector is
> stealing the disk.  If all you've got is a bunch of 8192-byte blocks,
> it's unlikely you can infer much about the contents.  You know the
> size of the relations and that's probably about it. 

Not necessarily. Our pages probably have enough predictable bytes to aid
cryptanalysis, compared to user data in a column which might not be very
predicable.


> If you've got individual values being encrypted, then there's more
> latitude to figure stuff out.  You can infer something about the
> length of particular values.  Perhaps you can find cases where the
> same encrypted value appears multiple times.

This completely depends on the encryption scheme you are using, and the
column level POLICY leaves that entirely up to you.

But in any case most encryption schemes use a random nonce (salt) to
ensure two identical strings do not encrypt to the same result. And
often the encrypted length is padded, so while you might be able to
infer short versus long, you would not usually be able to infer the
exact plaintext length.


> If there's a btree index, you know the ordering of the values under
> whatever ordering semantics apply to that index.  It's unclear to me
> how useful such information would be in practice or to what extent it
> might allow you to attack the underlying cryptography, but it seems
> like there might be cases where the information leakage is
> significant.  For example, suppose you're trying to determine which
> partially-encrypted record is that of Aaron Aardvark... or this guy: 
> https://en.wikipedia.org/wiki/Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff,_Sr.
Again, this only applies if your POLICY uses this type of encryption,
i.e. homomorphic encryption. If you use strong encryption you will not
be indexing those columns at all, which is pretty commonly the case.

> Recently, it was suggested to me that a use case for column-level
> encryption might be to prevent casual DBA snooping.  So, you'd want
> the data to appear in pg_dump output encrypted, because the DBA might
> otherwise look at it, but you wouldn't really be concerned about the
> threat of the DBA loading a hostile C module that would steal user
> keys and use them to decrypt all the data, because they don't care
> that much and would be fired if they were caught doing it.

Again completely dependent on the extension you use to do the encryption
for the input policy. The keys don't need to be stored with the data,
and the decryption can be transparent only for certain users or if
certain session variables exist which the DBA does not have access to.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
> Not necessarily. Our pages probably have enough predictable bytes to aid
> cryptanalysis, compared to user data in a column which might not be very
> predicable.

Really?  I would guess that the amount of entropy in a page is WAY
higher than in an individual column value.

> But in any case most encryption schemes use a random nonce (salt) to
> ensure two identical strings do not encrypt to the same result. And
> often the encrypted length is padded, so while you might be able to
> infer short versus long, you would not usually be able to infer the
> exact plaintext length.

Sure, that could be done, although it means that equality comparisons
must be done unencrypted.

> Again completely dependent on the extension you use to do the encryption
> for the input policy. The keys don't need to be stored with the data,
> and the decryption can be transparent only for certain users or if
> certain session variables exist which the DBA does not have access to.

Not arguing with that.  And to be clear, I'm not trying to attack your
proposal.  I'm just trying to have a discussion about advantages and
disadvantages of different approaches.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 06/18/2018 10:26 AM, Robert Haas wrote:
> On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
>> Not necessarily. Our pages probably have enough predictable bytes to aid
>> cryptanalysis, compared to user data in a column which might not be very
>> predicable.
> 
> Really?  I would guess that the amount of entropy in a page is WAY
> higher than in an individual column value.

It isn't about the entropy of the page overall, it is about the
predictability of specific bytes at specific locations on the pages. At
least as far as I understand it.

>> But in any case most encryption schemes use a random nonce (salt) to
>> ensure two identical strings do not encrypt to the same result. And
>> often the encrypted length is padded, so while you might be able to
>> infer short versus long, you would not usually be able to infer the
>> exact plaintext length.
> 
> Sure, that could be done, although it means that equality comparisons
> must be done unencrypted.

Sure. Typically equality comparisons are done on other unencrypted
attributes. Or if you need to do equality on encrypted columns, you can
store non-reversible cryptographic hashes in a separate column.

>> Again completely dependent on the extension you use to do the encryption
>> for the input policy. The keys don't need to be stored with the data,
>> and the decryption can be transparent only for certain users or if
>> certain session variables exist which the DBA does not have access to.
> 
> Not arguing with that.  And to be clear, I'm not trying to attack your
> proposal.  I'm just trying to have a discussion about advantages and
> disadvantages of different approaches.

Understood. Ultimately we might want both page-level encryption and
column level POLICY, as they are each useful for different use-cases.
Personally I believe the former is more generally useful than the
latter, but YMMV.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
>> Not necessarily. Our pages probably have enough predictable bytes to aid
>> cryptanalysis, compared to user data in a column which might not be very
>> predicable.

> Really?  I would guess that the amount of entropy in a page is WAY
> higher than in an individual column value.

Depending on the specifics of the encryption scheme, having some amount
of known (or guessable) plaintext may allow breaking the cipher, even
if much of the plaintext is not known.  This is cryptology 101, really.

At the same time, having to have a bunch of independently-decipherable
short field values is not real secure either, especially if they're known
to all be encrypted with the same key.  But what you know or can guess
about the plaintext in such cases would be target-specific, rather than
an attack that could be built once and used against any PG database.

            regards, tom lane

On 06/18/2018 10:52 AM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
>>> Not necessarily. Our pages probably have enough predictable bytes to aid
>>> cryptanalysis, compared to user data in a column which might not be very
>>> predicable.
> 
>> Really?  I would guess that the amount of entropy in a page is WAY
>> higher than in an individual column value.
> 
> Depending on the specifics of the encryption scheme, having some amount
> of known (or guessable) plaintext may allow breaking the cipher, even
> if much of the plaintext is not known.  This is cryptology 101, really.

Exactly

> At the same time, having to have a bunch of independently-decipherable
> short field values is not real secure either, especially if they're known
> to all be encrypted with the same key.  But what you know or can guess
> about the plaintext in such cases would be target-specific, rather than
> an attack that could be built once and used against any PG database.

Again is dependent on the specific solution for encryption. In some
cases you might do something like generate a single use random key,
encrypt the payload with that, encrypt the single use key with the
"global" key, append the two results and store.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 06/18/2018 05:06 PM, Joe Conway wrote:
> On 06/18/2018 10:52 AM, Tom Lane wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
>>>> Not necessarily. Our pages probably have enough predictable bytes to aid
>>>> cryptanalysis, compared to user data in a column which might not be very
>>>> predicable.
>>
>>> Really?  I would guess that the amount of entropy in a page is WAY
>>> higher than in an individual column value.
>>
>> Depending on the specifics of the encryption scheme, having some
>> amount of known (or guessable) plaintext may allow breaking the
>> cipher, even if much of the plaintext is not known. This is
>> cryptology 101, really.
> 
> Exactly
> 
>> At the same time, having to have a bunch of
>> independently-decipherable short field values is not real secure
>> either, especially if they're known to all be encrypted with the
>> same key. But what you know or can guess about the plaintext in
>> such cases would be target-specific, rather than an attack that
>> could be built once and used against any PG database.
> 
> Again is dependent on the specific solution for encryption. In some 
> cases you might do something like generate a single use random key, 
> encrypt the payload with that, encrypt the single use key with the 
> "global" key, append the two results and store.
> 

Yeah, I suppose we could even have per-page keys, for example.

One topic I haven't seen mentioned in this thread yet is indexes. That's 
a pretty significant side-channel, when built on encrypted columns. Even 
if the indexes are encrypted too, you can often deduce a lot of 
information from them.

So what's the plan here? Disallow indexes on encrypted columns? Index 
encypted values directly? Something else?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Jun 11, 2018 at 06:22:22PM +0900, Masahiko Sawada wrote:
> As per discussion at PGCon unconference, I think that firstly we need
> to discuss what threats we want to defend database data against. If

We call that a threat model.  There can be many threat models, of
course.

> user wants to defend against a threat that is malicious user who
> logged in OS or database steals an important data on datbase this
> design TDE would not help. Because such user can steal the data by
> getting a memory dump or by SQL. That is of course differs depending
> on system requirements or security compliance but what threats do you
> want to defend database data against? and why?

This design guards (somewhat) againts the threat of the storage theft
(e.g., because the storage is remote).  It's a fine threat model to
address, but it's also a lot easier to address in the filesystem or
device drivers -- there's no need to do this in PostgreSQL itself except
so as to support it on all platforms regardless of OS capabilities.

Note that unless the pg_catalog is protected against manipulation by
remote storage, then TDE for user tables might be possible to
compromise.  Like so: the attacker manipulates the pg_catalog to
escalate privelege in order to obtain the TDE keys.  This argues for
full database encryption, not just specific tables or columns.  But
again, this is for the threat model where the storage is the threat.

Another similar thread model is dump management, where dumps are sent
off-site where untrusted users might read them, or even edit them in the
hopes that they will be used for restores and thus compromise the
database.  This is most easily addressed by just encrypting the backups
externally to PG.

Threat models where client users are the threat are easily handled by
PG's permissions system.

I think any threat model where DBAs are not the threat is just not that
interesting to address with crypto within postgres itself...

Encryption to public keys for which postgres does not have private keys
would be one way to address DBAs-as-the-thread, but this is easily done
with an extension...  A small amount of syntactic sugar might help:

  CREATE ROLE "bar" WITH (PUBLIC KEY "...");

  CREATE TABLE foo (
    name TEXT PRIMARY KEY,
    payload TEXT ENCRYPTED TO ROLE "bar" BOUND TO name
  );

but this is just syntactic sugar, so not that valuable.  On the other
hand, just a bit of syntactic sugar can help tick a feature checkbox,
which might be very valuable for marketing reasons even if it's not
valuable for any other reason.

Note that encrypting the payload without a binding to the PK (or similar
name) is very dangerous!  So the encryption option would have to support
some way to indicate what other plaintext to bind in (here the "name"
column).

Note also that for key management reasons it would be necessary to be
able to write the payload as ciphertext rather than as to-be-encrypted
TEXT.

Lastly, for a symmetric encryption option one would need a remote oracle
to do the encryption, which seems rather complicated, but in some cases
may well perform faster.

Nico
-- 

On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote:
> BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this proposal.

> I'd like to hear opinions by security engineer.

Well, CBC makes sense, and since AES uses a 16 byte block size, you
would start with the initialization vector (IV) and run over the 8k page
512 times.  The IV can be any random value that is not repeated, and
does not need to be secret.

However, using the same IV for the entire table would mean that people
can detect if two pages in the same table contain the same data.  You
might care about that, or you might not.  It would prevent detection of
two _tables_ containing the same 8k page.  A more secure solution would
be to use a different IV for each 8k page.

The cleanest idea would be for the per-table IV to be stored per table,
but the IV used for each block to be a mixture of the table's IV and the
page's offset in the table.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote:
> On 06/11/2018 05:22 AM, Masahiko Sawada wrote:
> > As per discussion at PGCon unconference, I think that firstly we need
> > to discuss what threats we want to defend database data against.
> 
> Exactly. While certainly there is demand for encryption for the sake of
> "checking a box", different designs will defend against different
> threats, and we should be clear on which ones we are trying to protect
> against for any particular design.

Yep.  This slide covers the various encryption levels and the threats
they protect against:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=97

I do not have page-level encryption listed since that is not currently
possible with Postgres.

> > Also, if I understand correctly, at unconference session there also
> > were two suggestions about the design other than the suggestion by
> > Alexander: implementing TDE at column level using POLICY, and
> > implementing TDE at table-space level. The former was suggested by Joe
> > but I'm not sure the detail of that suggestion. I'd love to hear the
> > deal of that suggestion.
> 
> The idea has not been extensively fleshed out yet, but the thought was
> that we create column level POLICY, which would transparently apply some
> kind of transform on input and/or output. The transforms would
> presumably be expressions, which in turn could use functions (extension
> or builtin) to do their work. That would allow encryption/decryption,
> DLP (data loss prevention) schemes (masking, redacting), etc. to be
> applied based on the policies.

This is currently possible with stock Postgres as you can see from this
and the following slides:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=77

> This, in and of itself, would not address key management. There is
> probably a separate need for some kind of built in key management --
> perhaps a flexible way to integrate with external systems such as Vault
> for example, or maybe something self contained, or perhaps both. Or
> maybe key management is really tied into the separately discussed effort
> to create SQL VARIABLEs somehow.

I cover key management in this slide, and following:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=53

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote:
> >> Or
> >> maybe key management is really tied into the separately discussed effort
> >> to create SQL VARIABLEs somehow.
> > 
> > Could you elaborate on how key management is tied into SQL VARIABLEs?
> 
> Well, the key management probably is not, but the SQL VARIABLE might be
> where the key is stored for use.

I disagree.  I would need to understand how an extension actually helps
here, because it certainly limits flexibility compared to a shell
command.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote:
> figure stuff out.  You can infer something about the length of
> particular values.  Perhaps you can find cases where the same
> encrypted value appears multiple times.  If there's a btree index, you

Most encryption methods use a random initialization vector (IV) for each
encryption, e.g. pgp_sym_encrypt(), but length might allow this, as you
stated.

> know the ordering of the values under whatever ordering semantics
> apply to that index.  It's unclear to me how useful such information

I don't think an ordered index is possible, only indexing of encrypted
hashes, i.e. see this and the next slide:

    https://momjian.us/main/writings/crypto_hw_use.pdf#page=86

> would be in practice or to what extent it might allow you to attack
> the underlying cryptography, but it seems like there might be cases
> where the information leakage is significant.  For example, suppose
> you're trying to determine which partially-encrypted record is that of
> Aaron Aardvark... or this guy:
> https://en.wikipedia.org/wiki/Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff,_Sr.
> 
> Recently, it was suggested to me that a use case for column-level
> encryption might be to prevent casual DBA snooping.  So, you'd want
> the data to appear in pg_dump output encrypted, because the DBA might
> otherwise look at it, but you wouldn't really be concerned about the
> threat of the DBA loading a hostile C module that would steal user
> keys and use them to decrypt all the data, because they don't care
> that much and would be fired if they were caught doing it.

Yes, that is a benefit that is not possible with page-level encryption. 
It also encrypts the WAL and backups automatically;  see:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=97

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote:
> > At the same time, having to have a bunch of independently-decipherable
> > short field values is not real secure either, especially if they're known
> > to all be encrypted with the same key.  But what you know or can guess
> > about the plaintext in such cases would be target-specific, rather than
> > an attack that could be built once and used against any PG database.
> 
> Again is dependent on the specific solution for encryption. In some
> cases you might do something like generate a single use random key,
> encrypt the payload with that, encrypt the single use key with the
> "global" key, append the two results and store.

Even if they are encrypted with the same key, they use different
initialization vectors that are stored inside the encrypted payload, so
you really can't identify much except the length, as Robert stated.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
> On Mon, Jun 11, 2018 at 06:22:22PM +0900, Masahiko Sawada wrote:
> > As per discussion at PGCon unconference, I think that firstly we need
> > to discuss what threats we want to defend database data against. If
> 
> We call that a threat model.  There can be many threat models, of
> course.
> 
> > user wants to defend against a threat that is malicious user who
> > logged in OS or database steals an important data on datbase this
> > design TDE would not help. Because such user can steal the data by
> > getting a memory dump or by SQL. That is of course differs depending
> > on system requirements or security compliance but what threats do you
> > want to defend database data against? and why?
> 
> This design guards (somewhat) againts the threat of the storage theft
> (e.g., because the storage is remote).  It's a fine threat model to
> address, but it's also a lot easier to address in the filesystem or
> device drivers -- there's no need to do this in PostgreSQL itself except
> so as to support it on all platforms regardless of OS capabilities.
> 
> Note that unless the pg_catalog is protected against manipulation by
> remote storage, then TDE for user tables might be possible to
> compromise.  Like so: the attacker manipulates the pg_catalog to
> escalate privelege in order to obtain the TDE keys.  This argues for
> full database encryption, not just specific tables or columns.  But
> again, this is for the threat model where the storage is the threat.

Yes, one big problem with per-column encryption is that administrators
can silently delete data, though they can't add or modify it.

> Another similar thread model is dump management, where dumps are sent
> off-site where untrusted users might read them, or even edit them in the
> hopes that they will be used for restores and thus compromise the
> database.  This is most easily addressed by just encrypting the backups
> externally to PG.
> 
> Threat models where client users are the threat are easily handled by
> PG's permissions system.
> 
> I think any threat model where DBAs are not the threat is just not that
> interesting to address with crypto within postgres itself...


Yes, but in my analysis the only solution there is client-side
encryption:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=97

You might want to look at the earlier slides too.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote:
> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
> > Note that unless the pg_catalog is protected against manipulation by
> > remote storage, then TDE for user tables might be possible to
> > compromise.  Like so: the attacker manipulates the pg_catalog to
> > escalate privelege in order to obtain the TDE keys.  This argues for
> > full database encryption, not just specific tables or columns.  But
> > again, this is for the threat model where the storage is the threat.
> 
> Yes, one big problem with per-column encryption is that administrators
> can silently delete data, though they can't add or modify it.

They can also re-add ("replay") deleted values; this can only be
defeated by also binding TX IDs or alike in the ciphertext.  And if you
don't bind the encrypted values to the PKs then they can add any value
they've seen to different rows.

One can protect to some degree agains replay and reuse attacks, but
protecting against silent deletion is much harder.  Protecting against
the rows (or the entire DB) being restored at a past point in time is
even harder -- you quickly end up wanting Merkle hash/MAC trees and key
rotation, but this complicates everything and is performance killing.

> > I think any threat model where DBAs are not the threat is just not that
> > interesting to address with crypto within postgres itself...
> 
> Yes, but in my analysis the only solution there is client-side
> encryption:

For which threat model?

For threat models where the DBAs are not the threat there's no need for
client-side encryption: just encrypt the storage at the postgres
instance (with encrypting device drivers or -preferably- filesystems).

For threat models where the DBAs are the threat then yes, client-side
encryption works (or server-side encryption to public keys), but you
must still bind the encrypted values to the primary keys, and you must
provide integrity protection for as much data as possible -- see above.

Client-side crypto is hard to do well and still get decent performance.
So on the whole I think that crypto is a poor fit for the DBAs-are-the-
threat threat model.  It's better to reduce the number of DBAs/sysadmins
and audit all privileged (and, for good measure, unprivileged) access.

Client-side encryption, of course, wouldn't be a feature of PG..., as PG
is mostly a very smart server + very dumb clients.  The client could be
a lot smarter, for sure -- it could be a full-fledged RDBMS, it could
even be a postgres instance accessing the real server via FDW.

For example, libgda, the GNOME Data Assistant, IIRC, is a smart client
that uses SQLite3 to access remote resources via virtual table
extensions that function a lot like PG's FDW.  This works well because
SQLite3 is embeddable and light-weight.  PG wouldn't fit that bill as
well, but one could start a PG instance to proxy a remote one via FDW,
with crypto done in the proxy.

>     http://momjian.us/main/writings/crypto_hw_use.pdf#page=97
> 
> You might want to look at the earlier slides too.

I will, thanks.

Nico
-- 

On 06/20/2018 05:09 PM, Bruce Momjian wrote:
> On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote:
>> know the ordering of the values under whatever ordering semantics
>> apply to that index.  It's unclear to me how useful such information
> 
> I don't think an ordered index is possible, only indexing of encrypted
> hashes, i.e. see this and the next slide:

It is possible with homomorphic encryption -- whether we want to support
that in core is another matter.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 06/20/2018 05:05 PM, Bruce Momjian wrote:
> On Mon, Jun 18, 2018 at 08:29:32AM -0400, Joe Conway wrote:
>>>> Or
>>>> maybe key management is really tied into the separately discussed effort
>>>> to create SQL VARIABLEs somehow.
>>>
>>> Could you elaborate on how key management is tied into SQL VARIABLEs?
>>
>> Well, the key management probably is not, but the SQL VARIABLE might be
>> where the key is stored for use.
> 
> I disagree.  I would need to understand how an extension actually helps
> here, because it certainly limits flexibility compared to a shell
> command.

That flexibility the shell command gives you is also a huge hole from a
security standpoint.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Wed, Jun 20, 2018 at 06:06:56PM -0400, Joe Conway wrote:
> On 06/20/2018 05:09 PM, Bruce Momjian wrote:
> > On Mon, Jun 18, 2018 at 09:49:20AM -0400, Robert Haas wrote:
> >> know the ordering of the values under whatever ordering semantics
> >> apply to that index.  It's unclear to me how useful such information
> > 
> > I don't think an ordered index is possible, only indexing of encrypted
> > hashes, i.e. see this and the next slide:
> 
> It is possible with homomorphic encryption -- whether we want to support
> that in core is another matter.

It's also possible using DNSSEC NSEC3-style designs.

Nico
-- 

On 06/20/2018 05:03 PM, Bruce Momjian wrote:
> On Wed, Jun 13, 2018 at 09:20:58AM -0400, Joe Conway wrote:
>> The idea has not been extensively fleshed out yet, but the thought was
>> that we create column level POLICY, which would transparently apply some
>> kind of transform on input and/or output. The transforms would
>> presumably be expressions, which in turn could use functions (extension
>> or builtin) to do their work. That would allow encryption/decryption,
>> DLP (data loss prevention) schemes (masking, redacting), etc. to be
>> applied based on the policies.
> 
> This is currently possible with stock Postgres as you can see from this
> and the following slides:
> 
>     http://momjian.us/main/writings/crypto_hw_use.pdf#page=77

That is definitely not the same thing. A column level POLICY would apply
an input and output transform expression over the column transparently
to the database user. That transform might produce, for example, a
different output depending on the logged in user (certain user sees
entire field whereas other users see redacted or masked form, or certain
users get decrypted result while others don't).

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 06/20/2018 05:12 PM, Bruce Momjian wrote:
> On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote:
>>> At the same time, having to have a bunch of independently-decipherable
>>> short field values is not real secure either, especially if they're known
>>> to all be encrypted with the same key.  But what you know or can guess
>>> about the plaintext in such cases would be target-specific, rather than
>>> an attack that could be built once and used against any PG database.
>>
>> Again is dependent on the specific solution for encryption. In some
>> cases you might do something like generate a single use random key,
>> encrypt the payload with that, encrypt the single use key with the
>> "global" key, append the two results and store.
> 
> Even if they are encrypted with the same key, they use different
> initialization vectors that are stored inside the encrypted payload, so
> you really can't identify much except the length, as Robert stated.

The more you encrypt with a single key, the more fuel you give to the
person trying to solve for the key with cryptanalysis.

By encrypting only essentially random data (the single use keys,
generated with cryptographically strong random number generator) with
the "master key", and then encrypting the actual payloads (which are
presumably more predictable than the strong random single use keys), you
minimize the probability of someone cracking your master key and you
also minimize the damage caused by someone cracking one of the single
use keys.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Wed, Jun 20, 2018 at 06:19:40PM -0400, Joe Conway wrote:
> On 06/20/2018 05:12 PM, Bruce Momjian wrote:
> > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote:
> > Even if they are encrypted with the same key, they use different
> > initialization vectors that are stored inside the encrypted payload, so
> > you really can't identify much except the length, as Robert stated.

Definitely use different IVs, and don't reuse them (or use cipher modes
where IV reuse is not fatal).

> The more you encrypt with a single key, the more fuel you give to the
> person trying to solve for the key with cryptanalysis.

With modern 128-bit block ciphers in modern cipher modes you'd have to
encrypt enough data to make this not a problem.  On the other hand,
you'll still have other reasons to do key rotation.  Key rotation
ultimately means re-encrypting everything.  Getting all of this right is
very difficult.

So again, what's the threat model?  Because if it's sysadmins/DBAs
you're afraid of, there are better things to do.

Nico
-- 

From: Bruce Momjian [mailto:bruce@momjian.us]
> On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote:
> > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how
> to use the IV in CBC mode for this proposal.
> > I'd like to hear opinions by security engineer.
> 
> Well, CBC makes sense, and since AES uses a 16 byte block size, you
> would start with the initialization vector (IV) and run over the 8k page
> 512 times.  The IV can be any random value that is not repeated, and
> does not need to be secret.

XTS is faster and more secure.  XTS seems to be the standard now:

https://www.truecrypt71a.com/documentation/technical-details/encryption-scheme/
"c.Mode of operation: XTS, LRW (deprecated/legacy), CBC (deprecated/legacy)"

Microsoft Introduces AES-XTS to BitLocker in Windows 10 Version 1511
https://www.petri.com/microsoft-introduces-aes-xts-to-bitlocker-in-windows-10-version-1511


> However, using the same IV for the entire table would mean that people
> can detect if two pages in the same table contain the same data.  You
> might care about that, or you might not.  It would prevent detection of
> two _tables_ containing the same 8k page.  A more secure solution would
> be to use a different IV for each 8k page.
> 
> The cleanest idea would be for the per-table IV to be stored per table,
> but the IV used for each block to be a mixture of the table's IV and the
> page's offset in the table.

TrueCrypt uses the 8-byte sector number for the 16-byte tweak value for XTS when encrypting each sector.  Maybe we can
justuse the page number.
 


Regards
Takayuki Tsunakawa

On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams <nico@cryptonector.com> wrote:
> On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote:
>> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
>> > Note that unless the pg_catalog is protected against manipulation by
>> > remote storage, then TDE for user tables might be possible to
>> > compromise.  Like so: the attacker manipulates the pg_catalog to
>> > escalate privelege in order to obtain the TDE keys.  This argues for
>> > full database encryption, not just specific tables or columns.  But
>> > again, this is for the threat model where the storage is the threat.
>>
>> Yes, one big problem with per-column encryption is that administrators
>> can silently delete data, though they can't add or modify it.
>
> They can also re-add ("replay") deleted values; this can only be
> defeated by also binding TX IDs or alike in the ciphertext.  And if you
> don't bind the encrypted values to the PKs then they can add any value
> they've seen to different rows.

I think we could avoid it by implementations. If we implement
per-column encryption by putting all encrypted columns out to another
table like TOAST table and encrypting whole that external table then
we can do per-column encryption without such concerns. Also, that way
we can encrypt data when disk I/O even if we use per-column
encryption. It would get a better performance. A downside of this idea
is extra overhead to access encrypted column but it would be
predictable since we have TOAST.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Jun 21, 2018 at 10:05:41AM +0900, Masahiko Sawada wrote:
> On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams <nico@cryptonector.com> wrote:
> > On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote:
> >> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
> >> > Note that unless the pg_catalog is protected against manipulation by
> >> > remote storage, then TDE for user tables might be possible to
> >> > compromise.  Like so: the attacker manipulates the pg_catalog to
> >> > escalate privelege in order to obtain the TDE keys.  This argues for
> >> > full database encryption, not just specific tables or columns.  But
> >> > again, this is for the threat model where the storage is the threat.
> >>
> >> Yes, one big problem with per-column encryption is that administrators
> >> can silently delete data, though they can't add or modify it.
> >
> > They can also re-add ("replay") deleted values; this can only be
> > defeated by also binding TX IDs or alike in the ciphertext.  And if you
> > don't bind the encrypted values to the PKs then they can add any value
> > they've seen to different rows.
> 
> I think we could avoid it by implementations. If we implement
> per-column encryption by putting all encrypted columns out to another
> table like TOAST table and encrypting whole that external table then
> we can do per-column encryption without such concerns. Also, that way
> we can encrypt data when disk I/O even if we use per-column
> encryption. It would get a better performance. A downside of this idea
> is extra overhead to access encrypted column but it would be
> predictable since we have TOAST.

The case we were discussing was one where the threat model is that the
DBAs are the threat.  It is only in that case that the replay,
cut-n-paste, and silent deletion attacks are relevant.  Encrypting a
table, or the whole DB, on the server side, does nothing to protect
against that threat.

Never lose track of the threat model.

Nico
-- 

On Thu, Jun 21, 2018 at 2:53 PM, Nico Williams <nico@cryptonector.com> wrote:
> On Thu, Jun 21, 2018 at 10:05:41AM +0900, Masahiko Sawada wrote:
>> On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams <nico@cryptonector.com> wrote:
>> > On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote:
>> >> On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
>> >> > Note that unless the pg_catalog is protected against manipulation by
>> >> > remote storage, then TDE for user tables might be possible to
>> >> > compromise.  Like so: the attacker manipulates the pg_catalog to
>> >> > escalate privelege in order to obtain the TDE keys.  This argues for
>> >> > full database encryption, not just specific tables or columns.  But
>> >> > again, this is for the threat model where the storage is the threat.
>> >>
>> >> Yes, one big problem with per-column encryption is that administrators
>> >> can silently delete data, though they can't add or modify it.
>> >
>> > They can also re-add ("replay") deleted values; this can only be
>> > defeated by also binding TX IDs or alike in the ciphertext.  And if you
>> > don't bind the encrypted values to the PKs then they can add any value
>> > they've seen to different rows.
>>
>> I think we could avoid it by implementations. If we implement
>> per-column encryption by putting all encrypted columns out to another
>> table like TOAST table and encrypting whole that external table then
>> we can do per-column encryption without such concerns. Also, that way
>> we can encrypt data when disk I/O even if we use per-column
>> encryption. It would get a better performance. A downside of this idea
>> is extra overhead to access encrypted column but it would be
>> predictable since we have TOAST.
>
> The case we were discussing was one where the threat model is that the
> DBAs are the threat.  It is only in that case that the replay,
> cut-n-paste, and silent deletion attacks are relevant.  Encrypting a
> table, or the whole DB, on the server side, does nothing to protect
> against that threat.
>
> Never lose track of the threat model.
>

Understood.

>> On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams <nico@cryptonector.com> wrote:
>> So on the whole I think that crypto is a poor fit for the DBAs-are-the-
>> threat threat model.  It's better to reduce the number of DBAs/sysadmins
>> and audit all privileged (and, for good measure, unprivileged) access.

I agree with this. The in-database data encryption can defend mainly
the threat of storage theft and the threat of memory dump attack. I'm
sure this design had been proposed for the former purpose. If we want
to defend the latter we must encrypt data even on database memory. To
be honest, I'm not sure that there is needs in practice that is user
want to defend the memory dump attack. What user often needs is to
defend the threat of storage theft with minimum performance overhead.
It's known that client-side encryption or encryption on database
memory increase additional performance overheads. So it would be
better to have several ways to defend different threats as Joe
mentioned.

As long as we encrypt data transparently in database, both the
encryption of network between database server and client and
encryption of logical backups (e.g pg_dump) can be problem. For
network encryption we can use SSL for now but for logical backups we
need to address in other ways.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Jun 20, 2018 at 04:57:18PM -0500, Nico Williams wrote:
> On Wed, Jun 20, 2018 at 05:16:46PM -0400, Bruce Momjian wrote:
> > On Mon, Jun 18, 2018 at 12:29:57PM -0500, Nico Williams wrote:
> > > Note that unless the pg_catalog is protected against manipulation by
> > > remote storage, then TDE for user tables might be possible to
> > > compromise.  Like so: the attacker manipulates the pg_catalog to
> > > escalate privelege in order to obtain the TDE keys.  This argues for
> > > full database encryption, not just specific tables or columns.  But
> > > again, this is for the threat model where the storage is the threat.
> > 
> > Yes, one big problem with per-column encryption is that administrators
> > can silently delete data, though they can't add or modify it.
> 
> They can also re-add ("replay") deleted values; this can only be
> defeated by also binding TX IDs or alike in the ciphertext.  And if you

Yes, and if you bind TX IDs so you can detect loss, you effectively have
to serialize every transaction, which is going to kill performance.

> don't bind the encrypted values to the PKs then they can add any value
> they've seen to different rows.

Yep, you kind of have to add the primary key into the encrypted value.

> One can protect to some degree agains replay and reuse attacks, but
> protecting against silent deletion is much harder.  Protecting against
> the rows (or the entire DB) being restored at a past point in time is
> even harder -- you quickly end up wanting Merkle hash/MAC trees and key
> rotation, but this complicates everything and is performance killing.

Yep.

> > > I think any threat model where DBAs are not the threat is just not that
> > > interesting to address with crypto within postgres itself...
> > 
> > Yes, but in my analysis the only solution there is client-side
> > encryption:
> 
> For which threat model?
> 
> For threat models where the DBAs are not the threat there's no need for
> client-side encryption: just encrypt the storage at the postgres
> instance (with encrypting device drivers or -preferably- filesystems).

Agreed.

> For threat models where the DBAs are the threat then yes, client-side
> encryption works (or server-side encryption to public keys), but you
> must still bind the encrypted values to the primary keys, and you must
> provide integrity protection for as much data as possible -- see above.

Yep.

> Client-side crypto is hard to do well and still get decent performance.
> So on the whole I think that crypto is a poor fit for the DBAs-are-the-
> threat threat model.  It's better to reduce the number of DBAs/sysadmins
> and audit all privileged (and, for good measure, unprivileged) access.

Yeah, kind of.  There is the value of preventing accidental viewing of
the data by the DBA, and of course WAL and backup encryption are nice.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 21, 2018 at 04:49:34PM +0900, Masahiko Sawada wrote:
> >> On Thu, Jun 21, 2018 at 6:57 AM, Nico Williams <nico@cryptonector.com> wrote:
> >> So on the whole I think that crypto is a poor fit for the DBAs-are-the-
> >> threat threat model.  It's better to reduce the number of DBAs/sysadmins
> >> and audit all privileged (and, for good measure, unprivileged) access.
> 
> I agree with this. The in-database data encryption can defend mainly
> the threat of storage theft and the threat of memory dump attack. I'm
> sure this design had been proposed for the former purpose. If we want
> to defend the latter we must encrypt data even on database memory. To
> be honest, I'm not sure that there is needs in practice that is user
> want to defend the memory dump attack. What user often needs is to
> defend the threat of storage theft with minimum performance overhead.
> It's known that client-side encryption or encryption on database
> memory increase additional performance overheads. So it would be
> better to have several ways to defend different threats as Joe
> mentioned.

If you can view memory you can't really trust the server and have to do
encryption client-side.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Jun 20, 2018 at 06:19:40PM -0400, Joe Conway wrote:
> On 06/20/2018 05:12 PM, Bruce Momjian wrote:
> > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote:
> >>> At the same time, having to have a bunch of independently-decipherable
> >>> short field values is not real secure either, especially if they're known
> >>> to all be encrypted with the same key.  But what you know or can guess
> >>> about the plaintext in such cases would be target-specific, rather than
> >>> an attack that could be built once and used against any PG database.
> >>
> >> Again is dependent on the specific solution for encryption. In some
> >> cases you might do something like generate a single use random key,
> >> encrypt the payload with that, encrypt the single use key with the
> >> "global" key, append the two results and store.
> > 
> > Even if they are encrypted with the same key, they use different
> > initialization vectors that are stored inside the encrypted payload, so
> > you really can't identify much except the length, as Robert stated.
> 
> The more you encrypt with a single key, the more fuel you give to the
> person trying to solve for the key with cryptanalysis.
> 
> By encrypting only essentially random data (the single use keys,
> generated with cryptographically strong random number generator) with
> the "master key", and then encrypting the actual payloads (which are
> presumably more predictable than the strong random single use keys), you
> minimize the probability of someone cracking your master key and you
> also minimize the damage caused by someone cracking one of the single
> use keys.

Yeah, I have a slide about that too, and the previous and next slide:

    http://momjian.us/main/writings/crypto_hw_use.pdf#page=90

The more different keys you use the encrypt data, the more places you
have to store it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Jun 20, 2018 at 05:28:43PM -0500, Nico Williams wrote:
> On Wed, Jun 20, 2018 at 06:19:40PM -0400, Joe Conway wrote:
> > On 06/20/2018 05:12 PM, Bruce Momjian wrote:
> > > On Mon, Jun 18, 2018 at 11:06:20AM -0400, Joe Conway wrote:
> > > Even if they are encrypted with the same key, they use different
> > > initialization vectors that are stored inside the encrypted payload, so
> > > you really can't identify much except the length, as Robert stated.
> 
> Definitely use different IVs, and don't reuse them (or use cipher modes
> where IV reuse is not fatal).
> 
> > The more you encrypt with a single key, the more fuel you give to the
> > person trying to solve for the key with cryptanalysis.
> 
> With modern 128-bit block ciphers in modern cipher modes you'd have to
> encrypt enough data to make this not a problem.  On the other hand,
> you'll still have other reasons to do key rotation.  Key rotation
> ultimately means re-encrypting everything.  Getting all of this right is
> very difficult.
> 
> So again, what's the threat model?  Because if it's sysadmins/DBAs
> you're afraid of, there are better things to do.

Agreed.  Databases just don't match to the typical cryptographic
solutions and threat models.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 21, 2018 at 10:14:54AM -0400, Bruce Momjian wrote:
> On Wed, Jun 20, 2018 at 04:57:18PM -0500, Nico Williams wrote:
> > Client-side crypto is hard to do well and still get decent performance.
> > So on the whole I think that crypto is a poor fit for the DBAs-are-the-
> > threat threat model.  It's better to reduce the number of DBAs/sysadmins
> > and audit all privileged (and, for good measure, unprivileged) access.
> 
> Yeah, kind of.  There is the value of preventing accidental viewing of
> the data by the DBA, and of course WAL and backup encryption are nice.

One generally does not use crypto to prevent "accidental" viewing of
plaintext, but to provide real security relative to specific threats.

If you stop at encrypting values with no integrity protection for the
PKs, and no binding to TX IDs and such, you will indeed protect against
accidental viewing of the plaintext, but not against a determined
malicious insider.

Is that worthwhile?  Remember: you'll have to reduce and audit sysadmin
& DBA access anyways.

There is also the risk that users won't understand the limitations of
this sort of encryption feature and might get a false sense of security
from [mis]using it.

I'd want documentation to make it absolutely clear that such a feature
is only meant to reduce the risk of accidental viewing of plaintext by
DBAs and not a real security feature.

Nico
-- 

On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote:
> Issues on data encryption of PostgreSQL
> ==========
> Currently, in PostgreSQL, data encryption can be using pgcrypto Tool.
> However, it is inconvenient to use pgcrypto to encrypts data in some cases.
> 
> There are two significant inconveniences.
> 
> First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.

Not so.  VIEWs with INSTEAD OF triggers allow you to avoid this.

> Second, we must modify application program code much if we want to do
> database migration to PostgreSQL from other databases that is using
> TDE.

Not so.  See above.

However, I have at times been told that I should use SQL Server or
whatever because it has column encryption.  My answer is always that it
doesn't help (see my other posts on this thread), but from a business
perspective I understand the problem: the competition has a shiny (if
useless) feature XYZ, therefore we must have it also.

I'm not opposed to PG providing encryption features similar to the
competition's provided the documentation makes their false-sense-of-
security dangers clear.

Incidentally, PG w/ pgcrypto and FDW does provide everything one needs
to be able to implement client-side crypto:

 - use PG w/ FDW as a client-side proxy for the real DB
 - use pgcrypto in VIEWs with INSTEAD OF triggers in the proxy
 - access the DB via the proxy

Presto: transparent client-side crypto that protects against DBAs.  See
other posts about properly binding ciphertext and plaintext.

Protection against malicious DBAs is ultimately a very difficult thing
to get right -- if you really have DBAs as a threat and take that threat
seriously then you'll end up implementing a Merkle tree and performance
will go out the window.

> In these discussions, there were requirements necessary to support TDE in PostgreSQL.
> 
> 1) The performance overhead of encryption and decryption database data must be minimized
> 2) Need to support WAL encryption.
> 3) Need to support Key Management Service.

(2) and full database encryption could be done by the filesystem /
device drivers.  I think this is a much better answer than including
encryption in the DB just because it means not adding all that
complexity to PG, though it's not as portable as doing it in the DB (and
this may well be a winning argument).

What (3) looks like depends utterly on the threat model.  We must
discuss threat models first.

The threat models will drive the design, and (1) will drive some
trade-offs.

> Therefore, I'd like to propose the new design of TDE that deals with
> both above requirements.  Since this feature will become very large,
> I'd like to hear opinions from community before starting making the
> patch.

Any discussion of cryptographic applications should start with a
discussion of threat models.  This is not a big hurdle.

Nico
-- 

On Thu, Jun 21, 2018 at 12:12:40PM -0500, Nico Williams wrote:
> On Thu, Jun 21, 2018 at 10:14:54AM -0400, Bruce Momjian wrote:
> > On Wed, Jun 20, 2018 at 04:57:18PM -0500, Nico Williams wrote:
> > > Client-side crypto is hard to do well and still get decent performance.
> > > So on the whole I think that crypto is a poor fit for the DBAs-are-the-
> > > threat threat model.  It's better to reduce the number of DBAs/sysadmins
> > > and audit all privileged (and, for good measure, unprivileged) access.
> > 
> > Yeah, kind of.  There is the value of preventing accidental viewing of
> > the data by the DBA, and of course WAL and backup encryption are nice.
> 
> One generally does not use crypto to prevent "accidental" viewing of
> plaintext, but to provide real security relative to specific threats.
> 
> If you stop at encrypting values with no integrity protection for the
> PKs, and no binding to TX IDs and such, you will indeed protect against
> accidental viewing of the plaintext, but not against a determined
> malicious insider.
> 
> Is that worthwhile?  Remember: you'll have to reduce and audit sysadmin
> & DBA access anyways.
> 
> There is also the risk that users won't understand the limitations of
> this sort of encryption feature and might get a false sense of security
> from [mis]using it.
> 
> I'd want documentation to make it absolutely clear that such a feature
> is only meant to reduce the risk of accidental viewing of plaintext by
> DBAs and not a real security feature.

Agreed.  I can see from this discussion that we have a long way to go
before we can produce something clearly useful, but it will be worth it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 21, 2018 at 07:46:35PM -0400, Bruce Momjian wrote:
> Agreed.  I can see from this discussion that we have a long way to go
> before we can produce something clearly useful, but it will be worth it.

Let's start with a set of threat models then.  I'll go first:

1) storage devices as the threat
   a) theft of storage devices
   b) malicious storage device operators

2) malicious backup operators as the threat

3) malicious servers as the threat
   a) compromised servers
   b) insider threat -- rogue admins

4) malicious clients as the threat
   a) compromised clients
   b) insider threat

5) passive adversaries on the network as the threat

6) active adversaries on the network as the threat

7) adversaries on the same host as the server or client


Am I missing any?


For example, modern version control systems that use a Merkle hash tree
have malicious servers as part of their threat model.  Git clients, for
example, can detect non-fast-forward history changes upstream.

For another example, DNSSEC also provides protection against malicious
servers by authenticating not the servers but the _data_.  DNSSEC is a
useful case in point because it's effectively a key/value database that
stores somewhat relational data...


Clearly PG currently covers threat models 4 through 7:

 - passive adversaries on the network (addressed via TLS)
 - active adversaries on the network (addressed via TLS)
 - local adversaries (addressed by standard OS user process isolation)
 - malicious clients (addressed via authentication and authorization)

(1) and (2) can be covered externally:

 - protection against malicious storage or backup operators is trivial
   to provide: just use encrypting filesystems or device drivers, and
   encrypt backups using standard technologies.

One shortcoming of relying on OS functionality for protection against
malicious storage is that not all OSes may provide such functionality.
This could be an argument for implementing full, transparent encryption
for an entire DB in the postgres server.  Not a very compelling
argument, but that's just my opinion -- reasonable people could differ
on this.


PG also authenticates servers, but does nothing to authenticate the data
or functions of the server.  So while PG protects against illegitimate
server impersonators as well as TLS/GSS/SCRAM/... will afford, it does
not protect against rogue server admins nor against compromised servers.


That leaves (3) as the only threat model not covered.  It's also the
most challenging threat model to deal with.

Now, if you're going to protect against malicious servers (insiders)...

 - you can't let the server see any sensitive plaintext (must encrypt it)
 - which includes private/secret keys (the server can't have them, only
   the clients can)
 - you have to not only encrypt but provide integrity protection for
   ciphertext as well as unencrypted plaintext
 - decryption and integrity protection validation can only be done on
   the client (because only they have the necessary secrets!)

There are a lot of choices to make here that will greatly affect any
analysis of the security of the result.

A full analysis will inexorably lead to one conclusion: it's better to
just not have malicious servers (insiders), because if you really have
to defend against them then the only usable models of how to apply
cryptography to the problem are a) Git-like VCS, b) DNSSEC, and both are
rather heavy-duty for a general-purpose RDBMS.

So I think for (3) the best answer is to just not have that problem:
just reduce and audit admin access.

Still, if anyone wants to cover (3), I argue that PG gives you
everything you need right now: FDW and pgcrypto.  Just build a
solution where you have a PG server proxy that acts as a smart
client to untrusted servers:

                  +---------------+       +----------------+
   +--------+     |               |       |                |
   |        |     | postgres      |       | postgres       |
   |        |     | (proxy)       |       | (real server)  |
   | Client |---->|               |------>|                |
   |        |     |               |       |                |
   |        |     | (keys here)   |       | (no keys here) |
   +--------+     |               |       |                |
                  +---------------+       +----------------+

In the proxy use FDW (to talk to the real server) and VIEWs with INSTEAD
OF triggers to do all crypto transparently to the client.

Presto.  Transparent crypto right in your queries and DMLs.

But, you do have to get a number of choices right as to the crypto, and
chances are you won't provide integrity protection for the entire DB
(see above).

Nico
-- 

From: Nico Williams [mailto:nico@cryptonector.com]
> Let's start with a set of threat models then.  I'll go first:

Thank you so much for summarizing the current situation.  I'd appreciate it if you could write this on the PostgreSQL
wiki,when the discussion has settled somehow.
 


>  - local adversaries (addressed by standard OS user process isolation)

Does this also mean that we don't have to worry about the following?

* unencrypted data in the server process memory and core files
* passwords in .pgpass and recovery.conf (someone familiar with PCI DSS audit said this is a problem)
* user data in server logs


> One shortcoming of relying on OS functionality for protection against
> malicious storage is that not all OSes may provide such functionality.
> This could be an argument for implementing full, transparent encryption
> for an entire DB in the postgres server.  Not a very compelling
> argument, but that's just my opinion -- reasonable people could differ
> on this.

Yes, this is one reason I developed TDE in our product.  And in-database encryption allows optimization by encrypting
onlyuser data.
 


> So I think for (3) the best answer is to just not have that problem:
> just reduce and audit admin access.
> 
> Still, if anyone wants to cover (3), I argue that PG gives you
> everything you need right now: FDW and pgcrypto.  Just build a
> solution where you have a PG server proxy that acts as a smart
> client to untrusted servers:

Does sepgsql help?


Should a malfunctioning or buggy application be considered as as a threat?  That's what sql_firewall extension
addresses.

Regards
Takayuki Tsunakawa

On Fri, Jun 22, 2018 at 05:31:44AM +0000, Tsunakawa, Takayuki wrote:
> From: Nico Williams [mailto:nico@cryptonector.com]
> > Let's start with a set of threat models then.  I'll go first:
> 
> Thank you so much for summarizing the current situation.  I'd
> appreciate it if you could write this on the PostgreSQL wiki, when the
> discussion has settled somehow.

Sure, that's a good idea.

> >  - local adversaries (addressed by standard OS user process isolation)
> 
> Does this also mean that we don't have to worry about the following?
> 
> * unencrypted data in the server process memory and core files
> * passwords in .pgpass and recovery.conf (someone familiar with PCI
>   DSS audit said this is a problem)
> * user data in server logs

Short of using things like Intel SGX or homomorphic encryption, I don't
think we can do anything about plaintext in memory -- at some point it
has to be there, therefore it is as vulnerable as the host OS makes it.

Users can always run only the one postgres instance and nothing else
(and no other non-admin users) on the host to reduce local attack
surface to zero.

So, yes, I think this flavor of local vulnerability should be out of
scope for PG.

> > One shortcoming of relying on OS functionality for protection against
> > malicious storage is that not all OSes may provide such functionality.
> > This could be an argument for implementing full, transparent encryption
> > for an entire DB in the postgres server.  Not a very compelling
> > argument, but that's just my opinion -- reasonable people could differ
> > on this.
> 
> Yes, this is one reason I developed TDE in our product.  And
> in-database encryption allows optimization by encrypting only user
> data.

I understand this motivation.  I wouldn't reject this out of hand, even
though I'm not exactly interested either.

Can you keep the impact on the codebase isolated and limited, and the
performance impact when disabled to zero?

> > So I think for (3) the best answer is to just not have that problem:
> > just reduce and audit admin access.
> > 
> > Still, if anyone wants to cover (3), I argue that PG gives you
> > everything you need right now: FDW and pgcrypto.  Just build a
> > solution where you have a PG server proxy that acts as a smart
> > client to untrusted servers:
> 
> Does sepgsql help?

Any functionality in PG that allows DBAs to manage storage, sessions,
..., without having to see table data will help.  It doesn't ahve to be
tied to trusted OS functionality.

I've not looked at SEP [0] so I don't know if it helps.  I would prefer
that PG simply have native functionality to allow this sort of
separation -- as I'm not a DBA, I don't really know if PG has this.

[0] https://wiki.postgresql.org/wiki/SEPostgreSQL_SELinux_Overview

> Should a malfunctioning or buggy application be considered as as a
> threat?  That's what sql_firewall extension addresses.

I suppose so, yes.

Nico
-- 

On Fri, Jun 22, 2018 at 2:31 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:
> From: Nico Williams [mailto:nico@cryptonector.com]
>> Let's start with a set of threat models then.  I'll go first:
>
> Thank you so much for summarizing the current situation.  I'd appreciate it if you could write this on the PostgreSQL
wiki,when the discussion has settled somehow.
 
>
>
>>  - local adversaries (addressed by standard OS user process isolation)
>
> Does this also mean that we don't have to worry about the following?
>
> * unencrypted data in the server process memory and core files
> * passwords in .pgpass and recovery.conf (someone familiar with PCI DSS audit said this is a problem)
> * user data in server logs
>
>
>> One shortcoming of relying on OS functionality for protection against
>> malicious storage is that not all OSes may provide such functionality.
>> This could be an argument for implementing full, transparent encryption
>> for an entire DB in the postgres server.  Not a very compelling
>> argument, but that's just my opinion -- reasonable people could differ
>> on this.
>
> Yes, this is one reason I developed TDE in our product.  And in-database encryption allows optimization by encrypting
onlyuser data.
 
>

Me too. In-database encryption is helpful in practice. I think 1) and
2) seem to cover the thread models which the data encryption in
database needs to defend.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 21/06/18 21:43, Nico Williams wrote:
> On Fri, May 25, 2018 at 08:41:46PM +0900, Moon, Insung wrote:
>> Issues on data encryption of PostgreSQL
>> ==========
>> Currently, in PostgreSQL, data encryption can be using pgcrypto Tool.
>> However, it is inconvenient to use pgcrypto to encrypts data in some cases.
>>
>> There are two significant inconveniences.
>>
>> First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.
> Not so.  VIEWs with INSTEAD OF triggers allow you to avoid this.
>
>> Second, we must modify application program code much if we want to do
>> database migration to PostgreSQL from other databases that is using
>> TDE.
> Not so.  See above.
>
> However, I have at times been told that I should use SQL Server or
> whatever because it has column encryption.  My answer is always that it
> doesn't help (see my other posts on this thread), but from a business
> perspective I understand the problem: the competition has a shiny (if
> useless) feature XYZ, therefore we must have it also.
>
> I'm not opposed to PG providing encryption features similar to the
> competition's provided the documentation makes their false-sense-of-
> security dangers clear.
>
> Incidentally, PG w/ pgcrypto and FDW does provide everything one needs
> to be able to implement client-side crypto:
>
>   - use PG w/ FDW as a client-side proxy for the real DB
>   - use pgcrypto in VIEWs with INSTEAD OF triggers in the proxy
>   - access the DB via the proxy

     Sounds a bit hackish, but it could work. I doubt however the 
acceptance of a solution like this, given the number of "moving parts" 
and operational complexity associated with it.


>
> Presto: transparent client-side crypto that protects against DBAs.  See
> other posts about properly binding ciphertext and plaintext.
>
> Protection against malicious DBAs is ultimately a very difficult thing
> to get right -- if you really have DBAs as a threat and take that threat
> seriously then you'll end up implementing a Merkle tree and performance
> will go out the window.
>
>> In these discussions, there were requirements necessary to support TDE in PostgreSQL.
>>
>> 1) The performance overhead of encryption and decryption database data must be minimized
>> 2) Need to support WAL encryption.
>> 3) Need to support Key Management Service.
> (2) and full database encryption could be done by the filesystem /
> device drivers.  I think this is a much better answer than including
> encryption in the DB just because it means not adding all that
> complexity to PG, though it's not as portable as doing it in the DB (and
> this may well be a winning argument).
>
> What (3) looks like depends utterly on the threat model.  We must
> discuss threat models first.
>
> The threat models will drive the design, and (1) will drive some
> trade-offs.
>
>> Therefore, I'd like to propose the new design of TDE that deals with
>> both above requirements.  Since this feature will become very large,
>> I'd like to hear opinions from community before starting making the
>> patch.
> Any discussion of cryptographic applications should start with a
> discussion of threat models.  This is not a big hurdle.


     You already mentioned that there are also "business criteria" to 
consider here, and they are important. And there are even more to 
consider. For instance, cases where (1) and even (2) under your proposed 
threat model cannot be fixed by external (device/filesystem) encryption. 
Consider for example the managed database services provided by the cloud 
vendors. While (all?) of them provide transparent disk encryption, are 
they trust-able? Do business want to rely on their encryption scheme, 
key management, and how they respond from requests to hand off 
encryption keys? I believe self-contained solutions are very worth, also 
because of this.


     Álvaro

-- 

Alvaro Hernandez


-----------
OnGres

On 11/06/18 12:22, Masahiko Sawada wrote:
> On Fri, May 25, 2018 at 8:41 PM, Moon, Insung
> <Moon_Insung_i3@lab.ntt.co.jp> wrote:
>> Hello Hackers,
>>
>> This propose a way to develop "Table-level" Transparent Data Encryption (TDE) and Key Management Service (KMS)
supportin
 
>> PostgreSQL.
>>
>>
>> Issues on data encryption of PostgreSQL
>> ==========
>> Currently, in PostgreSQL, data encryption can be using pgcrypto Tool.
>> However, it is inconvenient to use pgcrypto to encrypts data in some cases.
>>
>> There are two significant inconveniences.
>>
>> First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.
>> Second, we must modify application program code much if we want to do database migration to PostgreSQL from other
databasesthat is
 
>> using TDE.
>>
>> To resolved these inconveniences, many users want to support TDE.
>> There have also been a few proposals, comments, and questions to support TDE in the PostgreSQL community.
>>
>> However, currently PostgreSQL does not support TDE, so in development community, there are discussions whether it's
necessaryto
 
>> support TDE or not.
>>
>> In these discussions, there were requirements necessary to support TDE in PostgreSQL.
>>
>> 1) The performance overhead of encryption and decryption database data must be minimized
>> 2) Need to support WAL encryption.
>> 3) Need to support Key Management Service.
>>
>> Therefore, I'd like to propose the new design of TDE that deals with both above requirements.
>> Since this feature will become very large, I'd like to hear opinions from community before starting making the
patch.
>>
>> First, my proposal is table-level TDE which is that user can specify tables begin encrypted.
>> Indexes, TOAST table and WAL associated with the table that enables TDE are also encrypted.
>>
>> Moreover, I want to support encryption for large object as well.
>> But I haven't found a good way for it so far. So I'd like to remain it as future TODO.
>>
>> My proposal has five characteristics features of "table-level TDE".
>>
>> 1) Buffer-level data encryption and decryption
>> 2) Per-table encryption
>> 3) 2-tier encryption key management
>> 4) Working with external key management services(KMS)
>> 5) WAL encryption
>>
>> Here are more details for each items.
>>
>>
>> 1. Buffer-level data encryption and decryption
>> ==================
>> Transparent data encryption and decryption accompany by storage operation
>> With ordinally way like using pgcrypto, the biggest problem with encrypted data is the performance overhead of
decryptingthe data
 
>> each time the run to queries.
>>
>> My proposal is to encrypt and decrypt data when performing DISK I/O operation to minimize performance overhead.
>> Therefore, the data in the shared memory layer is unencrypted so that performance overhead can minimize.
>>
>> With this design, data encryption/decryption implementations can be developed by modifying the codes of the storage
andbuffer
 
>> manager modules,
>> which are responsible for performing DISK I/O operation.
>>
>>
>> 2. Per-table encryption
>> ==================
>> User can enable TDE per table as they want.
>> I introduce new storage parameter "encryption_enabled" which enables TDE at table-level.
>>
>>      // Generate  the encryption table
>>         CREATE TABLE foo WITH ( ENCRYPTION_ENABLED = ON );
>>
>>      // Change to the non-encryption table
>>         ALTER TABLE foo SET ( ENCRYPTION_ENABLED = OFF );
>>
>> This approach minimizes the overhead for tables that do not require encryption options.
>> For tables that enable TDE, the corresponding table key will be generated with random values, and it's stored into
thenew system
 
>> catalog after being encrypted by the master key.
>>
>> BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this
proposal.
>> I'd like to hear opinions by security engineer.
>>
>>
>> 3. 2-tier encryption key management
>> ==================
>> when it comes time to change cryptographic keys, there is a performance overhead to decryption and re-encryption to
alldata.
 
>>
>> To solve this problem we employee 2-tier encryption.
>> 2-tier encryption is All table keys can be stored in the database cluster after being encrypted by the master key,
Andmaster keys
 
>> must be stored at external of PostgreSQL.
>>
>> Therefore, without master key, it is impossible to decrypt the table key. Thus, It is impossible to decrypt the
databasedata.
 
>>
>> When changing the key, it's not necessary to re-encrypt for all data.
>> We use the new master key only to decrypt and re-encrypt the table key, these operations for minimizing the
performanceoverhead.
 
>>
>> For table keys, all TDE-enabled tables have different table keys.
>> And for master key, all database have different master keys. Table keys are encrypted by the master key of its own
database.
>> For WAL encryption, we have another cryptographic key. WAL-key is also encrypted by a master key, but it is shared
acrossthe
 
>> database cluster.
>>
>>
>> 4. Working with external key management services(KMS)
>> ==================
>> A key management service is an integrated approach for generating, fetching and managing encryption keys for key
control.
>> They may cover all aspects of security from the secure generation of keys, secure storing keys, and secure fetching
keysup to
 
>> encryption key handling.
>> Also, various types of KMSs are provided by many companies, and users can choose them.
>>
>> Therefore I would like to manage the master key using KMS.
>> Also, my proposal is to create callback APIs(generate_key, fetch_key, store_key) in the form of a plug-in so that
userscan use many
 
>> types of KMS as they want.
>>
>> In KMIP protocol and most KMS manage keys by string IDs. We can get keys by key ID from KMS.
>> So in my proposal, all master keys are distinguished by its ID, called "master key ID".
>> The master key ID is made, for example, using the database oid and a sequence number, like <OID>_<SeqNo>. And they
aremanaged in
 
>> PostgreSQL.
>>
>> When database startup, all master key ID is loaded to shared memory, and they are protected by LWLock.
>>
>> When it comes time to rotate the master keys, run this query.
>>
>>          ALTER SYSTEM ROTATION MASTER KEY;
>>
>> In this query, the master key is rotated with the following step.
>> 1. Generate new master key,
>> 2. Change master key IDs and emit corresponding WAL
>> 3. Re-encrypt all table keys on its database
>>
>> Also during checkpoint, master key IDs on shared memory become a permanent condition.
>>
>>
>> 5. WAL encryption
>> ==================
>> If we encrypt all WAL records, performance overhead can be significant.
>> Therefore, this proposes a method to encrypt only WAL record excluding WAL header when writing WAL on the WAL
buffer,instead of
 
>> encrypting a whole WAL record.
>> WAL encryption key is generated separately when the TDE-enabled table is created the first time. We use 2-tier
encryptionfor WAL
 
>> encryption as well.
>> So, when it comes time to rotate the WAL encryption key, run this query.
>>
>>          ALTER SYSTEM ROTATION WAL KEY;
>>
>> Next, I will explain how to encrypt WAL.
>>
>> To do this operation, I add a flag to WAL header which indicates whether the subsequent WAL data is encrypted or
not.
>>
>> Then, when we write WAL for encryption table we write "encrypted" WAL on WAL buffer layer.
>>
>> In recovery, we read WAL header and check the flag of encryption, and judges whether WAL must be decrypted.
>> In the case of PITR, we use WAL key ID in the backup file.
>>
>> With this approach, the performance overhead of writing and reading the WAL for unencrypted tables would be almost
thesame as
 
>> before.
>>
>>

     I may have missed part of the conversation and/or this may be a 
naïve question, but what about pg_stats? I guess data should be 
encrypted there too, and I wonder how this would affect the query 
planner and how it could decrypt this information. Also would a separate
key be used for the stats?


     Thanks,

     Álvaro


-- 

Alvaro Hernandez


-----------
OnGres

On Mon, Jul 02, 2018 at 06:56:34PM +0300, Alvaro Hernandez wrote:
> On 21/06/18 21:43, Nico Williams wrote:
> >Incidentally, PG w/ pgcrypto and FDW does provide everything one needs
> >to be able to implement client-side crypto:
> >
> >  - use PG w/ FDW as a client-side proxy for the real DB
> >  - use pgcrypto in VIEWs with INSTEAD OF triggers in the proxy
> >  - access the DB via the proxy
> 
> Sounds a bit hackish, but it could work. I doubt however the acceptance
> of a solution like this, given the number of "moving parts" and operational
> complexity associated with it.

Well, you could use SQLite3 instead as the client.  Like how GDA does
it.

I do wish there was a libpostgres -- a light-weight postgres for running
an in-process RDBMS in applications without having to have a separate
set of server processes.  That would work really well for a use case
like this one where you're really going to be using FDW to access the
real data store.

If your objection is to an RDBMS in the application accessing real data
via FDW, well, see all the commentary about threat models.  You really
can't protect against DBAs without client-side crypto (and lots of bad
trade-offs).  You can do the crypto in the application, but you start to
lose the power of SQL.  Anyways, I don't think client-side crypto is the
best answer to the DBA threat -- access reduction + auditing is easier
and better.

In any case, spinning up a postgres instance just for this use case is
easy because it wouldn't have any persistent state to keep locally.

> >Any discussion of cryptographic applications should start with a
> >discussion of threat models.  This is not a big hurdle.
> 
> You already mentioned that there are also "business criteria" to
> consider here, and they are important. And there are even more to consider.

The threat model *is* a business criterion.  What are the threats you
want to protect against?  Why aren't you interested in these other
threats?  These are *business* questions.

Of course, one has to understand the issues, including intangibles.  For
example, reputation risk is a huge business concern, but as an
intangible it's easy to miss.  People have a blind spot for intangibles.

> For instance, cases where (1) and even (2) under your proposed threat model
> cannot be fixed by external (device/filesystem) encryption. Consider for
> example the managed database services provided by the cloud vendors. While
> (all?) of them provide transparent disk encryption, are they trust-able? Do

Databases won't be your only cloud security issue.  At some point you're
either using things like SGX or, while you wait for practical, high-
performance homomorphic cryptgraphic computing, you'll settle for using
the power of contracts and contract law -- contract law is a *business*
tool.

At some point there's not much difference between an insider you can
fire and an insider at a vendor who can fire them (and which vendor you
can fire as well), especially when you can use contract law to get the
vendor to do special things for you, like show you how they do things,
reduce admin access, let you audit them, and so on.

> business want to rely on their encryption scheme, key management, and how
> they respond from requests to hand off encryption keys? I believe
> self-contained solutions are very worth, also because of this.

I don't, because client-side crypto to deal with this is just so very
unsatisfactory.  Besides, what happens when first you move the DB into
the cloud and put a lot of effort into client-side crypto, then you move
the clients into the same cloud too?

And anyways, what was proposed by OP is *server-side* crypto, which
clearly does not help at all in this case.

Nico
-- 

On Mon, Jul 02, 2018 at 06:22:46PM +0900, Masahiko Sawada wrote:
> On Fri, Jun 22, 2018 at 2:31 PM, Tsunakawa, Takayuki
> <tsunakawa.takay@jp.fujitsu.com> wrote:
> > From: Nico Williams [mailto:nico@cryptonector.com]
> >
> >> One shortcoming of relying on OS functionality for protection against
> >> malicious storage is that not all OSes may provide such functionality.
> >> This could be an argument for implementing full, transparent encryption
> >> for an entire DB in the postgres server.  Not a very compelling
> >> argument, but that's just my opinion -- reasonable people could differ
> >> on this.
> >
> > Yes, this is one reason I developed TDE in our product.  And
> > in-database encryption allows optimization by encrypting only user
> > data.

You're likely getting some things terribly wrong.  E.g., integrity
protection.  Most likely you're getting a false sense of security.

> Me too. In-database encryption is helpful in practice. I think 1) and
> 2) seem to cover the thread models which the data encryption in
> database needs to defend.

Yes, but piecemeal encryption seems like a bad idea to me.

Nico
-- 

On Tue, Jul 3, 2018 at 7:16 AM, Nico Williams <nico@cryptonector.com> wrote:
> On Mon, Jul 02, 2018 at 06:22:46PM +0900, Masahiko Sawada wrote:
>> On Fri, Jun 22, 2018 at 2:31 PM, Tsunakawa, Takayuki
>> <tsunakawa.takay@jp.fujitsu.com> wrote:
>> > From: Nico Williams [mailto:nico@cryptonector.com]
>> >
>> >> One shortcoming of relying on OS functionality for protection against
>> >> malicious storage is that not all OSes may provide such functionality.
>> >> This could be an argument for implementing full, transparent encryption
>> >> for an entire DB in the postgres server.  Not a very compelling
>> >> argument, but that's just my opinion -- reasonable people could differ
>> >> on this.
>> >
>> > Yes, this is one reason I developed TDE in our product.  And
>> > in-database encryption allows optimization by encrypting only user
>> > data.
>
> You're likely getting some things terribly wrong.  E.g., integrity
> protection.  Most likely you're getting a false sense of security.
>
>> Me too. In-database encryption is helpful in practice. I think 1) and
>> 2) seem to cover the thread models which the data encryption in
>> database needs to defend.
>
> Yes, but piecemeal encryption seems like a bad idea to me.
>

What do you mean by "piecemeal encryption"? Is it not-whole database
encryption such as per-table or per-tablespace? If so could you please
elaborate on the reason why you think so?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Dear Antonin Houska.

> -----Original Message-----
> From: Antonin Houska [mailto:ah@cybertec.at]
> Sent: Tuesday, May 29, 2018 3:23 PM
> To: Moon, Insung
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
>
> Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:
>
> This patch seems to implement some of the features you propose, especially encryption of buffers and WAL. I recommend
> you to check so that no effort is
> duplicated:

Yes. encrypting / decrypting between Buffer <-> Disk is the same architecture.
But, this idea is not to encrypt all table, thinks to minimize the performance overhead, only encrypting to necessary
tables(including Xlog). 

Thank you and Best regards.
Moon.

>
> > [4] Recently discussed mail
> >
> > https://www.postgresql.org/message-id/CA%2BCSw_tb3bk5i7if6inZFc3yyf%2B
> > 9HEVNTy51QFBoeUk7UE_V%3Dw%40mail.gmail.com
>
>
>
> --
> Antonin Houska
> Cybertec Schönig & Schönig GmbH
> Gröhrmühlgasse 26, A-2700 Wiener Neustadt
> Web: https://www.cybertec-postgresql.com

Dear Aleksander Alekseev.

> -----Original Message-----
> From: Aleksander Alekseev [mailto:a.alekseev@postgrespro.ru]
> Sent: Thursday, May 31, 2018 10:33 PM
> To: Moon, Insung
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> Hello Moon,
> 
> I promised to email links to the articles I mentioned during your talk on the PGCon Unconference to this thread.
Here
> they are:
> 
> * http://cryptowiki.net/index.php?title=Order-preserving_encryption
> * https://en.wikipedia.org/wiki/Homomorphic_encryption
> 
> Also I realized that I was wrong regarding encryption of the indexes since they will be encrypted on the block level
the
> same way the heap will be.

Sorry. I did not explain correctly in PGCon.
Yes. this idea is encrypting at the block level as you said, there is probably not a big problem with index
encryption.
I will testing with PoC later an Index Encryption.

Thank you and Best regards.
Moon.


> 
> --
> Best regards,
> Aleksander Alekseev

Dear Masahiko Sawada.

> -----Original Message-----
> From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]
> Sent: Monday, June 11, 2018 6:22 PM
> To: Moon, Insung
> Cc: PostgreSQL-development; Joe Conway
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
>
> On Fri, May 25, 2018 at 8:41 PM, Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:
> > Hello Hackers,
> >
> > This propose a way to develop "Table-level" Transparent Data
> > Encryption (TDE) and Key Management Service (KMS) support in PostgreSQL.
> >
> >
> > Issues on data encryption of PostgreSQL ========== Currently, in
> > PostgreSQL, data encryption can be using pgcrypto Tool.
> > However, it is inconvenient to use pgcrypto to encrypts data in some cases.
> >
> > There are two significant inconveniences.
> >
> > First, if we use pgcrypto to encrypt/decrypt data, we must call pgcrypto functions everywhere we encrypt/decrypt.
> > Second, we must modify application program code much if we want to do
> > database migration to PostgreSQL from other databases that is using TDE.
> >
> > To resolved these inconveniences, many users want to support TDE.
> > There have also been a few proposals, comments, and questions to support TDE in the PostgreSQL community.
> >
> > However, currently PostgreSQL does not support TDE, so in development
> > community, there are discussions whether it's necessary to support TDE or not.
> >
> > In these discussions, there were requirements necessary to support TDE in PostgreSQL.
> >
> > 1) The performance overhead of encryption and decryption database data
> > must be minimized
> > 2) Need to support WAL encryption.
> > 3) Need to support Key Management Service.
> >
> > Therefore, I'd like to propose the new design of TDE that deals with both above requirements.
> > Since this feature will become very large, I'd like to hear opinions from community before starting making the
patch.
> >
> > First, my proposal is table-level TDE which is that user can specify tables begin encrypted.
> > Indexes, TOAST table and WAL associated with the table that enables TDE are also encrypted.
> >
> > Moreover, I want to support encryption for large object as well.
> > But I haven't found a good way for it so far. So I'd like to remain it as future TODO.
> >
> > My proposal has five characteristics features of "table-level TDE".
> >
> > 1) Buffer-level data encryption and decryption
> > 2) Per-table encryption
> > 3) 2-tier encryption key management
> > 4) Working with external key management services(KMS)
> > 5) WAL encryption
> >
> > Here are more details for each items.
> >
> >
> > 1. Buffer-level data encryption and decryption ==================
> > Transparent data encryption and decryption accompany by storage
> > operation With ordinally way like using pgcrypto, the biggest problem
> > with encrypted data is the performance overhead of decrypting the data each time the run to queries.
> >
> > My proposal is to encrypt and decrypt data when performing DISK I/O operation to minimize performance overhead.
> > Therefore, the data in the shared memory layer is unencrypted so that performance overhead can minimize.
> >
> > With this design, data encryption/decryption implementations can be
> > developed by modifying the codes of the storage and buffer manager
> > modules, which are responsible for performing DISK I/O operation.
> >
> >
> > 2. Per-table encryption
> > ==================
> > User can enable TDE per table as they want.
> > I introduce new storage parameter "encryption_enabled" which enables TDE at table-level.
> >
> >     // Generate  the encryption table
> >        CREATE TABLE foo WITH ( ENCRYPTION_ENABLED = ON );
> >
> >     // Change to the non-encryption table
> >        ALTER TABLE foo SET ( ENCRYPTION_ENABLED = OFF );
> >
> > This approach minimizes the overhead for tables that do not require encryption options.
> > For tables that enable TDE, the corresponding table key will be
> > generated with random values, and it's stored into the new system catalog after being encrypted by the master key.
> >
> > BTW, I want to support CBC mode encryption[3]. However, I'm not sure how to use the IV in CBC mode for this
proposal.
> > I'd like to hear opinions by security engineer.
> >
> >
> > 3. 2-tier encryption key management
> > ==================
> > when it comes time to change cryptographic keys, there is a performance overhead to decryption and re-encryption to
> all data.
> >
> > To solve this problem we employee 2-tier encryption.
> > 2-tier encryption is All table keys can be stored in the database
> > cluster after being encrypted by the master key, And master keys must be stored at external of PostgreSQL.
> >
> > Therefore, without master key, it is impossible to decrypt the table key. Thus, It is impossible to decrypt the
database
> data.
> >
> > When changing the key, it's not necessary to re-encrypt for all data.
> > We use the new master key only to decrypt and re-encrypt the table key, these operations for minimizing the
performance
> overhead.
> >
> > For table keys, all TDE-enabled tables have different table keys.
> > And for master key, all database have different master keys. Table keys are encrypted by the master key of its own
database.
> > For WAL encryption, we have another cryptographic key. WAL-key is also
> > encrypted by a master key, but it is shared across the database cluster.
> >
> >
> > 4. Working with external key management services(KMS)
> > ================== A key management service is an integrated approach
> > for generating, fetching and managing encryption keys for key control.
> > They may cover all aspects of security from the secure generation of
> > keys, secure storing keys, and secure fetching keys up to encryption key handling.
> > Also, various types of KMSs are provided by many companies, and users can choose them.
> >
> > Therefore I would like to manage the master key using KMS.
> > Also, my proposal is to create callback APIs(generate_key, fetch_key,
> > store_key) in the form of a plug-in so that users can use many types of KMS as they want.
> >
> > In KMIP protocol and most KMS manage keys by string IDs. We can get keys by key ID from KMS.
> > So in my proposal, all master keys are distinguished by its ID, called "master key ID".
> > The master key ID is made, for example, using the database oid and a
> > sequence number, like <OID>_<SeqNo>. And they are managed in PostgreSQL.
> >
> > When database startup, all master key ID is loaded to shared memory, and they are protected by LWLock.
> >
> > When it comes time to rotate the master keys, run this query.
> >
> >         ALTER SYSTEM ROTATION MASTER KEY;
> >
> > In this query, the master key is rotated with the following step.
> > 1. Generate new master key,
> > 2. Change master key IDs and emit corresponding WAL 3. Re-encrypt all
> > table keys on its database
> >
> > Also during checkpoint, master key IDs on shared memory become a permanent condition.
> >
> >
> > 5. WAL encryption
> > ==================
> > If we encrypt all WAL records, performance overhead can be significant.
> > Therefore, this proposes a method to encrypt only WAL record excluding
> > WAL header when writing WAL on the WAL buffer, instead of encrypting a whole WAL record.
> > WAL encryption key is generated separately when the TDE-enabled table
> > is created the first time. We use 2-tier encryption for WAL encryption as well.
> > So, when it comes time to rotate the WAL encryption key, run this query.
> >
> >         ALTER SYSTEM ROTATION WAL KEY;
> >
> > Next, I will explain how to encrypt WAL.
> >
> > To do this operation, I add a flag to WAL header which indicates whether the subsequent WAL data is encrypted or
not.
> >
> > Then, when we write WAL for encryption table we write "encrypted" WAL on WAL buffer layer.
> >
> > In recovery, we read WAL header and check the flag of encryption, and judges whether WAL must be decrypted.
> > In the case of PITR, we use WAL key ID in the backup file.
> >
> > With this approach, the performance overhead of writing and reading
> > the WAL for unencrypted tables would be almost the same as before.
> >
> >
> > ==================
> > I'd like to discuss the design before starting making any change of code.
> > After a more discussion I want to make a PoC.
> > Feedback and suggestion are very welcome.
> >
> > Finally, thank you initial design input for Masahiko Sawada.
> >
> > Thank you.
> >
> > [1] What does TDE mean?
> >     > https://en.wikipedia.org/wiki/Transparent_Data_Encryption
> >
> > [2] What does KMS mean?
> >     >
> > https://en.wikipedia.org/wiki/Key_management#Key_Management_System
> >
> > [3] What does CBC-Mode mean?
> >     > https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
> >
> > [4] Recently discussed mail
> >
> > https://www.postgresql.org/message-id/CA%2BCSw_tb3bk5i7if6inZFc3yyf%2B
> > 9HEVNTy51QFBoeUk7UE_V%3Dw%40mail.gmail.com
> >
> >
>
> As per discussion at PGCon unconference, I think that firstly we need to discuss what threats we want to defend
database
> data against. If user wants to defend against a threat that is malicious user who logged in OS or database steals an
important
> data on datbase this design TDE would not help. Because such user can steal the data by getting a memory dump or by
SQL.
> That is of course differs depending on system requirements or security compliance but what threats do you want to
defend
> database data against? and why?

Yes. I'm Checking to the requirement 3.4 of PCI-DSS.
This requirement is a refer to encrypting stored data.
And idea does not protect data against memory dump(include coredump).
If required for an encryption of memory layer, I'll recheck to this idea.
And I will do a little more research on any enterprise requirement on encryption data.

>
> Also, if I understand correctly, at unconference session there also were two suggestions about the design other than
the
> suggestion by
> Alexander: implementing TDE at column level using POLICY, and implementing TDE at table-space level. The former was
suggested
> by Joe but I'm not sure the detail of that suggestion. I'd love to hear the deal of that suggestion. The latter was
suggested
> by Tsunakawa-san.
> Have you considered that?

First, thank you for Joe and Tsunakawa-san.
I'm thinking of table-level encrypting, but I'll try to find the best way through this discussion.

>
> You mentioned that encryption of temporary data for query processing and large objects are still under the
consideration.
> But other than them you should consider the temporary data generated by other subsystems such as reorderbuffer and
transition
> table as well.

Yes. Encryption of temporary data and large objects and anymore is considered essential.
In this case, I have not yet decided how to encrypt temporary data. I'll make PoC patch, and find to how to encryption
oftemporary data. 

Thank you and Best regards.
Moon.



>
> Regards,
>
> --
> Masahiko Sawada
> NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center

Dear Tomas Vondra.

> -----Original Message-----
> From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
> Sent: Wednesday, June 13, 2018 10:15 PM
> To: Moon, Insung; pgsql-hackers@postgresql.org
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> Hi,
> 
> On 05/25/2018 01:41 PM, Moon, Insung wrote:
> > Hello Hackers,
> >
> > ...
> >
> > BTW, I want to support CBC mode encryption[3]. However, I'm not sure
> > how to use the IV in CBC mode for this proposal. I'd like to hear
> > opinions by security engineer.
> >
> 
> I'm not a cryptographer either, but this is exactly where you need a prior discussion about the threat models -
there
> are a couple of chaining modes, each with different weaknesses.
> 

Thank you for your advice.
First, I'm researched to more security problem and found that CBC mode is an not safe encryption mode.
Later, when I'll create a PoC, using to GCM or XTS encryption mode.
And this time I know for using the same IV is dangerous, and I'm doing some more research on this.

Thank you and Best regards.
Moon.


> FWIW it may also matter if data_checksums are enabled, because that may prevent malleability attacks affecting of
the
> modes. Assuming active attacker (with the ability to modify the data files) is part of the threat model, of course.
> 
> regards
> 
> --
> Tomas Vondra                  http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Jul 03, 2018 at 07:28:42PM +0900, Masahiko Sawada wrote:
> On Tue, Jul 3, 2018 at 7:16 AM, Nico Williams <nico@cryptonector.com> wrote:
> > Yes, but piecemeal encryption seems like a bad idea to me.
> 
> What do you mean by "piecemeal encryption"? Is it not-whole database
> encryption such as per-table or per-tablespace? If so could you please
> elaborate on the reason why you think so?

I mean that encrypting some columns only, or some tables only, has
integrity protection issues.  See earlier posts in this thread.

Encrypting the whole DB has no such problems, assuming you're doing the
crypto correctly anyways.  But for full DB encryption it's easier to
leave the crypto to the filesystem or device drivers.  (If the devices
are physically in the host and cannot be removed easily, then FDE at the
device works well too.)

Nico
-- 

Dear Tomas Vondra.

> -----Original Message-----
> From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
> Sent: Wednesday, June 13, 2018 10:03 PM
> To: Masahiko Sawada; Moon, Insung
> Cc: PostgreSQL-development; Joe Conway
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> On 06/11/2018 11:22 AM, Masahiko Sawada wrote:
> > On Fri, May 25, 2018 at 8:41 PM, Moon, Insung
> > <Moon_Insung_i3@lab.ntt.co.jp> wrote:
> >> Hello Hackers,
> >>
> >> This propose a way to develop "Table-level" Transparent Data
> >> Encryption (TDE) and Key Management Service (KMS) support in
> >> PostgreSQL.
> >>
> >> ...
> >
> > As per discussion at PGCon unconference, I think that firstly we need
> > to discuss what threats we want to defend database data against.
> > If user wants to defend against a threat that is malicious user who
> > logged in OS or database steals an important data on datbase this
> > design TDE would not help. Because such user can steal the data by
> > getting a memory dump or by SQL. That is of course differs depending
> > on system requirements or security compliance but what threats do you
> > want to defend database data against? and why?
> >
> 
> I do agree with this - a description of the threat model needs to be part of the design discussion, otherwise it's
not
> possible to compare it to alternative solutions (e.g. full-disk encryption using LUKS or using existing privilege
controls
> and/or RLS).
> 
> TDE was proposed/discussed repeatedly in the past, and every time it died exactly because it was not very clear
which
> issue it was attempting to solve.
> 
> Let me share some of the issues mentioned as possibly addressed by TDE (I'm not entirely sure TDE actually solves
them,
> I'm just saying those were mentioned in previous discussions):
> 
> 1) enterprise requirement - Companies want in-database encryption, for various reasons (because "enterprise
solution"
> or something).

Yes. I do not know clearly about enterprise encryption requirements.
Typically, identified the requirements for encryption of PCI-DSS and posted these ideas.(Storage encryptoin)
Therefore, according to your opinion, I will more try to research of the enterprise encryption requirements.

> 
> 2) like FDE, but OS/filesystem independent - Same config on any OS and filesystem, which may make maintenance
easier.
> 
> 3) does not require special OS/filesystem setup - Does not require help from system adminitrators, setup of LUKS
devices
> or whatever.

Yes. We can use disk encryption like LUKS at Linux, but it does not apply to all OS's, so I'm proposed TDE.

> 
> 4) all filesystem access (basebackups/rsync) is encrypted anyway
> 
> 5) solves key management (the main challenge with pgcrypto)

In fact, it is the biggest worry about key management.
First, I think of 2-tier encryption as I wrote in my idea, and I am thinking of using KMS for management to master
key.
However, I am also worried about security problems when I managed of table key and master key.
Therefore, I want to more discuss of Key Management and develop KMS simultaneously with TDE.


Thank you and Best regards.
Moon.


> 
> 6) allows encrypting only some of the data (tables, columns) to minimize performance impact
> 
> IMHO it makes sense to have TDE even if it provides the same "security"
> as disk-level encryption, assuming it's more convenient to setup/use from the database.
> 
> > Also, if I understand correctly, at unconference session there also
> > were two suggestions about the design other than the suggestion by
> > Alexander: implementing TDE at column level using POLICY, and
> > implementing TDE at table-space level. The former was suggested by Joe
> > but I'm not sure the detail of that suggestion. I'd love to hear the
> > deal of that suggestion. The latter was suggested by Tsunakawa-san.
> > Have you considered that?
> >
> > You mentioned that encryption of temporary data for query processing
> > and large objects are still under the consideration. But other than
> > them you should consider the temporary data generated by other
> > subsystems such as reorderbuffer and transition table as well.
> >
> 
> The severity of those limitations is likely related to the threat model.
> I don't think encrypting temporary data would be a big problem, assuming you know which key to use.
> 
> regards
> 
> --
> Tomas Vondra                  http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Dear Takayuki Tsunakawa.

> -----Original Message-----
> From: Tsunakawa, Takayuki [mailto:tsunakawa.takay@jp.fujitsu.com]
> Sent: Thursday, June 14, 2018 9:58 AM
> To: 'Tomas Vondra'; Moon, Insung; pgsql-hackers@postgresql.org
> Subject: RE: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> > From: Tomas Vondra [mailto:tomas.vondra@2ndquadrant.com]
> > On 05/25/2018 01:41 PM, Moon, Insung wrote:
> > > BTW, I want to support CBC mode encryption[3]. However, I'm not sure
> > > how to use the IV in CBC mode for this proposal. I'd like to hear
> > > opinions by security engineer.
> > >
> >
> > I'm not a cryptographer either, but this is exactly where you need a
> > prior discussion about the threat models - there are a couple of
> > chaining modes, each with different weaknesses.
> Our products uses XTS, which recent FDE software like BitLocker and TrueCrypt uses instead of CBC.
> 
> https://en.wikipedia.org/wiki/Disk_encryption_theory#XTS
> 
> "According to SP 800-38E, "In the absence of authentication or access control, XTS-AES provides more protection than
the
> other approved confidentiality-only modes against unauthorized manipulation of the encrypted data.""

Thank your for your advice!

Yes. I found that CBC is not safe at this time.
So let's use XTS mode or GCM mode as you mentioned.

Thank you and Best regards.
Moon.

> 
> 
> 
> > FWIW it may also matter if data_checksums are enabled, because that
> > may prevent malleability attacks affecting of the modes. Assuming
> > active attacker (with the ability to modify the data files) is part of
> > the threat model, of course.
> 
> Encrypt the page after embedding its checksum value.  If a malicious attacker modifies a page on disk, then the
decrypted
> page would be corrupt anyway, which can be detected by checksum.
> 
> 
> Regards
> Takayuki Tsunakawa
> 

Dear Joe.

> -----Original Message-----
> From: Joe Conway [mailto:mail@joeconway.com]
> Sent: Monday, June 18, 2018 9:30 PM
> To: Masahiko Sawada
> Cc: Moon, Insung; PostgreSQL-development
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> On 06/14/2018 12:19 PM, Masahiko Sawada wrote:
> > On Wed, Jun 13, 2018 at 10:20 PM, Joe Conway <mail@joeconway.com> wrote:
> >> The idea has not been extensively fleshed out yet, but the thought
> >> was that we create column level POLICY, which would transparently
> >> apply some kind of transform on input and/or output. The transforms
> >> would presumably be expressions, which in turn could use functions
> >> (extension or builtin) to do their work. That would allow
> >> encryption/decryption, DLP (data loss prevention) schemes (masking,
> >> redacting), etc. to be applied based on the policies.
> >
> > Which does this design encrypt data on, buffer or both buffer and
> > disk?
> 
> 
> The point of the design is simply to provide a mechanism for input and output transformation, not to provide the
transform
> function itself.
> 
> How you use that transformation would be entirely up to you, but if you were providing an encryption transform on
input
> the data would be encrypted both buffer and disk.
> 
> > And does this design (per-column encryption) aim to satisfy something
> > specific security compliance?
> 
> 
> Again, entirely up to you and dependent on what type of transformation you provide. If, for example you provided
input
> encryption and output decryption based on some in memory session variable key, that would be essentially TDE and
would
> satisfy several common sets of compliance requirements.
> 
> 
> >> This, in and of itself, would not address key management. There is
> >> probably a separate need for some kind of built in key management --
> >> perhaps a flexible way to integrate with external systems such as
> >> Vault for example, or maybe something self contained, or perhaps both.
> >
> > I agree to have a flexible way in order to address different
> > requirements. I thought that having a GUC parameter to which we store
> > a shell command to get encryption key is enough but considering
> > integration with various key managements seamlessly I think that we
> > need to have APIs for key managements. (fetching key, storing key,
> > generating key etc)
> 
> 
> I don't like the idea of yet another path for arbitrary shell code execution. An API for extension code would be
preferable.

Thank you for your advice on key management.
In fact, it was a big worry how to implement key management.
Basically, we will look at the rules of KMIP, and I'll try to create an extension API that can mostly work with KMS.

and I have a question.
You said do not like the idea of another path for arbitrary shell code execution, is there any special reason?
For example, I think usability to specify a path for shell code to use several KMSs, Is there a potential security
issue?

Thank you and Best regards.
Moon.


> 
> 
> >> Or
> >> maybe key management is really tied into the separately discussed
> >> effort to create SQL VARIABLEs somehow.
> >
> > Could you elaborate on how key management is tied into SQL VARIABLEs?
> 
> Well, the key management probably is not, but the SQL VARIABLE might be where the key is stored for use.
> 
> Joe
> 
> --
> Crunchy Data - http://crunchydata.com
> PostgreSQL Support for Secure Enterprises Consulting, Training, & Open Source Development

Dear Tom Lane.

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Monday, June 18, 2018 11:52 PM
> To: Robert Haas
> Cc: Joe Conway; Masahiko Sawada; Moon, Insung; PostgreSQL-development
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> 
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
> >> Not necessarily. Our pages probably have enough predictable bytes to
> >> aid cryptanalysis, compared to user data in a column which might not
> >> be very predicable.
> 
> > Really?  I would guess that the amount of entropy in a page is WAY
> > higher than in an individual column value.
> 
> Depending on the specifics of the encryption scheme, having some amount of known (or guessable) plaintext may allow
breaking
> the cipher, even if much of the plaintext is not known.  This is cryptology 101, really.
> 
> At the same time, having to have a bunch of independently-decipherable short field values is not real secure either,
especially
> if they're known to all be encrypted with the same key.  But what you know or can guess about the plaintext in such
cases
> would be target-specific, rather than an attack that could be built once and used against any PG database.

Yes. If there is known to guessable data of encrypted data, maybe there is a  possibility of decrypting the encrypted
data.

But would it be safe to use an additional encryption mode such as GCM or XFS to solve this problem?
(Do not use the same IV)

Thank you and Best regards.
Moon.


> 
>             regards, tom lane

On Tue, Jul 3, 2018 at 5:37 PM Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:

Dear Tom Lane.

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Monday, June 18, 2018 11:52 PM
> To: Robert Haas
> Cc: Joe Conway; Masahiko Sawada; Moon, Insung; PostgreSQL-development
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
>
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
> >> Not necessarily. Our pages probably have enough predictable bytes to
> >> aid cryptanalysis, compared to user data in a column which might not
> >> be very predicable.
>
> > Really?  I would guess that the amount of entropy in a page is WAY
> > higher than in an individual column value.
>
> Depending on the specifics of the encryption scheme, having some amount of known (or guessable) plaintext may allow breaking
> the cipher, even if much of the plaintext is not known.  This is cryptology 101, really.
>
> At the same time, having to have a bunch of independently-decipherable short field values is not real secure either, especially
> if they're known to all be encrypted with the same key.  But what you know or can guess about the plaintext in such cases
> would be target-specific, rather than an attack that could be built once and used against any PG database.

Yes. If there is known to guessable data of encrypted data, maybe there is a  possibility of decrypting the encrypted data.

But would it be safe to use an additional encryption mode such as GCM or XFS to solve this problem?
(Do not use the same IV)

Thank you and Best regards.
Moon.


>
>                       regards, tom lane

Hi Moon,

Have you done progress on that patch? I am thinking to work on the project and found that you are already working on it. The last message is almost six months old. I want to check with you that are you still working on that, if yes I can help on that by reviewing the patch etc. If you are not working on that anymore, can you share your done work (if possible)?

--

Ibrar Ahmed

-- 
Hans-Jürgen Schönig
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: https://www.cybertec-postgresql.com

Dear Ibrar Ahmed.

From: Ibrar Ahmed [mailto:ibrar.ahmad@gmail.com]
Sent: Thursday, February 07, 2019 4:09 AM
To: Moon, Insung
Cc: Tom Lane; PostgreSQL-development
Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)


On Tue, Jul 3, 2018 at 5:37 PM Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:
Dear Tom Lane.

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Monday, June 18, 2018 11:52 PM
> To: Robert Haas
> Cc: Joe Conway; Masahiko Sawada; Moon, Insung; PostgreSQL-development
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
>
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
> >> Not necessarily. Our pages probably have enough predictable bytes to
> >> aid cryptanalysis, compared to user data in a column which might not
> >> be very predicable.
>
> > Really?  I would guess that the amount of entropy in a page is WAY
> > higher than in an individual column value.
>
> Depending on the specifics of the encryption scheme, having some amount of known (or guessable) plaintext may allow
breaking
> the cipher, even if much of the plaintext is not known.  This is cryptology 101, really.
>
> At the same time, having to have a bunch of independently-decipherable short field values is not real secure either,
especially
> if they're known to all be encrypted with the same key.  But what you know or can guess about the plaintext in such
cases
> would be target-specific, rather than an attack that could be built once and used against any PG database.

> > Yes. If there is known to guessable data of encrypted data, maybe there is a  possibility of decrypting the
encrypteddata. 
> >
> > But would it be safe to use an additional encryption mode such as GCM or XFS to solve this problem?
> > (Do not use the same IV)
> > Thank you and Best regards.
> > Moon.
>
> >
> >                       regards, tom lane





> Hi Moon,
>
> Have you done progress on that patch? I am thinking to work on the project and found that you are already working on
it.The last message is almost six months old. I want to check with you that are you still working on that, if yes I can
helpon that by reviewing the patch etc. If you are not working on that anymore, can you share your done work (if
possible)?
> --
> Ibrar Ahmed

We are currently developing for TDE and integration KMS.
So, We will Also be prepared to start a new discussion with the PoC patch as soon as possible.

At currently, we have changed the development direction of a per-Tablespace unit by per-table
Also, currently researching how to associate with KMIP protocol related to the encryption key for integration with KMS.
We talked about this in the Unconference session of PGConf.ASIA,
And a week ago, we talked about the development direction of TDE and integration with KMS at FOSDEM PGDAY[1].

We will soon provide PoC with new discussions.

Regards.

[1] TRANSPARENT DATA ENCRYPTION IN POSTGRESQL AND INTEGRATION WITH KEY MANAGEMENT SERVICES

https://www.postgresql.eu/events/fosdem2019/schedule/session/2307-transparent-data-encryption-in-postgresql-and-integration-with-key-management-services/

On Thu, Feb 7, 2019 at 9:27 AM Moon, Insung
<Moon_Insung_i3@lab.ntt.co.jp> wrote:
>
> Dear Ibrar Ahmed.
>
> From: Ibrar Ahmed [mailto:ibrar.ahmad@gmail.com]
> Sent: Thursday, February 07, 2019 4:09 AM
> To: Moon, Insung
> Cc: Tom Lane; PostgreSQL-development
> Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
>
>
> On Tue, Jul 3, 2018 at 5:37 PM Moon, Insung <Moon_Insung_i3@lab.ntt.co.jp> wrote:
> Dear Tom Lane.
>
> > -----Original Message-----
> > From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> > Sent: Monday, June 18, 2018 11:52 PM
> > To: Robert Haas
> > Cc: Joe Conway; Masahiko Sawada; Moon, Insung; PostgreSQL-development
> > Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)
> >
> > Robert Haas <robertmhaas@gmail.com> writes:
> > > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway <mail@joeconway.com> wrote:
> > >> Not necessarily. Our pages probably have enough predictable bytes to
> > >> aid cryptanalysis, compared to user data in a column which might not
> > >> be very predicable.
> >
> > > Really?  I would guess that the amount of entropy in a page is WAY
> > > higher than in an individual column value.
> >
> > Depending on the specifics of the encryption scheme, having some amount of known (or guessable) plaintext may allow
breaking
> > the cipher, even if much of the plaintext is not known.  This is cryptology 101, really.
> >
> > At the same time, having to have a bunch of independently-decipherable short field values is not real secure
either,especially 
> > if they're known to all be encrypted with the same key.  But what you know or can guess about the plaintext in such
cases
> > would be target-specific, rather than an attack that could be built once and used against any PG database.
>
> > > Yes. If there is known to guessable data of encrypted data, maybe there is a  possibility of decrypting the
encrypteddata. 
> > >
> > > But would it be safe to use an additional encryption mode such as GCM or XFS to solve this problem?
> > > (Do not use the same IV)
> > > Thank you and Best regards.
> > > Moon.
> >
> > >
> > >                       regards, tom lane
>
>
>
>
>
> > Hi Moon,
> >
> > Have you done progress on that patch? I am thinking to work on the project and found that you are already working
onit. The last message is almost six months old. I want to check with you that are you still working on that, if yes I
canhelp on that by reviewing the patch etc. If you are not working on that anymore, can you share your done work (if
possible)?
> > --
> > Ibrar Ahmed
>
> We are currently developing for TDE and integration KMS.
> So, We will Also be prepared to start a new discussion with the PoC patch as soon as possible.
>
> At currently, we have changed the development direction of a per-Tablespace unit by per-table
> Also, currently researching how to associate with KMIP protocol related to the encryption key for integration with
KMS.
> We talked about this in the Unconference session of PGConf.ASIA,
> And a week ago, we talked about the development direction of TDE and integration with KMS at FOSDEM PGDAY[1].
>
> We will soon provide PoC with new discussions.
>
> Regards.
>
> [1] TRANSPARENT DATA ENCRYPTION IN POSTGRESQL AND INTEGRATION WITH KEY MANAGEMENT SERVICES
>
https://www.postgresql.eu/events/fosdem2019/schedule/session/2307-transparent-data-encryption-in-postgresql-and-integration-with-key-management-services/
>

Let me share the details of progress and current state.

As the our presentation slides describes I've written the PoC code for
transparent encryption that uses 2-tier key architecture and has the
key rotation feature. We've been discussed the design database
transparent encryption on -hackers so far and we found a good design
and implementation. I will share them with our research results. But I
think the design of integration of PostgreSQL with key management
services(KMS) is more controvertible.

For integration with KMS, I'm going to propose to add generic key
management APIs to PostgreSQL core so that it can communicate with
KMSs supporting different interfaces and protocols and can get the
master key (of 2-tier key architecture) from them. Users can choose a
key management plugin according to their enviornment.

The integration of PostgreSQL with KMS should be separated patch from
the TDE patch and we think that TDE can be done first. But at least
it's essential to provide a way to get the master key from an external
location. Therefore as the first step we can propose the basic
components of TDE with a simple interface to get the master key from
KMS rather than supporting full key management APIs. The basic
components of TDE that we're going to propose are:

* Transparent encryption at a layer between shared buffer and OS page cache
* Per tablespaces encryption
* 2-tier key architecture
* Key rotation
* System catalogs and temporary files encryption

WAL encryption will follow as an additional feature.

The simple interface to get the master key is a GUC parameter that can
store the shell command, say get_encryption_key_command. As its names
suggests, the command is used for only getting the master key, never
be used for removal and registration.

The slides explains about TDE feature in details but doesn't about KMS
much. So let me share a rough idea of using TDE in combination with
KMS.

2-Tier Key Architecture and Key Generation
=================================

In our design, we use 2-tier key architecuter which uses two types
keys: one master key and multiple data encryption keys. As the slides
explains details, the benefit of this architecture is the fast key
rotation. When the key rotation the data to re-encrypt is only data
encryption keys.

Key Generation Number is an integer value starting from 1, using for
identifying the master key. It's initialized at initdb time and
incremented whenever the master key is changed (i.g. key rotation).
For each key generation number we have multiple data encryption keys
associated with tablespaces. The current key generation number is
written to checkpoint records. When starting up, the startup process
executes the shell command set in get_encryption_key_command GUC
parameter with a key generation number.

For example, we can set something like get_encryption_key_command =
'/bin/sh get_key_from_kms.sh %g', where '%g' is replaced with the
current key generation number and where 'get_key_from_kms.sh' is an
arbitary shell script to get the master key from a KMS. I assume that
the master keys on KMS can be identified by its ID. So DBA generates a
master key identified by the key ID in a arbitary form on KMS
beforehand and the get_encryption_key_command has to crafts the key ID
in the same manner and pass to the KMS. The master key we got is
written to stdout.

Therefore, the contract between PostgreSQL and user is,
* User must prepare the master key identified by an unique key ID in advance
* The shell command crafts the key ID in the same form as key ID on KMS.
* User must remove old keys from KMS if necessary (because there is no
interface other than getting the master key)

Initial Setup and Recovery
====================

Since the user data could be encrypted we need the data encryption
keys and the master key even during recovery. The
get_encryption_key_command will be executed by the startup process
with the key generation number written in the checkpoint record, and
stores the master key to the shared memory.

For example, if we crafts the master key ID in the form of 'ABC_<key
generation number>', the operation steps from initdb to recovery will
be followings.

1. User creates the master key of first generation with ID 'ABC_1' on KMS
2. User executes initdb and sets get_encryption_key_command = '/bin/sh
get_key_from_kms.sh %g' in postgresql.conf
3. Start PostgreSQL
    3-1. If transparent encryption is disabled or there is no
encrypted data on database go to step #4
    3-2. The startup process executes '/bin/sh get_key_from_kms.sh 1'
because the current(initial) key generation is 1
    3-3. The get_key_from_kms.sh crafts the key ID 'ABC_1' in the same
form and get the master key from KMS
    3-4. If failed, raise a FATAL error
    3-5. Store the master key to the shared memory
    3-6. If there is data encryption key, decrypt them using the master key
4. Recovery starts

To make sure that we got the correct master key we can save the hash
value of master key on the database cluster and compare them.

Key Rotation
===========

When user requests key rotation (via SQL command or function), the
backends execute get_encryption_key_command with the new key
generation number. It re-encrypts all existing data encryption keys
with the new master key and increments the current key generation
number. Similar to initialization time, we need to prepare the new
master key on the KMS before executing the key rotation.

So for example, the operation steps will be like;

1. Create the second generation master key with key ID 'ABC_2' on KMS
2. Execute key rotation on PostgreSQL (calling
pg_rotate_encryption_key() function)
    3-1. The backends execute '/bin/sh get_key_from_kms.sh 2', where 2
is the next key generation number
    3-2. It crafts the key ID 'ABC_2' in the same manner and gets the
new master key from KMS
    3-3. If failed, raise an error
    3-4. Re-encrypt data encryption keys using the new master key
    3-5. Increment the current key generation to 2

Of course some lockings are required here.

Integration with KMS
================

This above design has some restrictions on administration but might be
enough for a few use case. But I think that these inconviniences will
go way if we had KMS integration. Since KMIP supports some protocol
for key management such as key registration and key removal, the key
management plugin will be responsible for registrating the master key
and getting it using the key ID generated in an unified form. So what
user need to do are setting up KMS and setting up the key management
plugin. User no longer needs to create and remove the master key
manually.

Other use case of integrating with KMS
==============================

BTW we can have not only internal key management interfaces for TDE
feature but also SQL interface for existing use-cases such as using
pgcrypto. Currently we need to pass the password to encryption and
decryption functions.

SELECT decrypt(data, 'sercret-key', 'aes') FROM ...;

The password will be logged to the server log when log_statement =
'all'. But with KMSs it would become,

SELECT decrypt(data, get_encryption_key('keyid'), 'aes') FROM ...;

where get_encryption_key() function gets the encryption key from KMS
via the loaded plugin. The key string never be output to the server
logs.


We're still researching the details of KMIP and key managements APIs.
Will share the updates. Feedback is very welcome and we're open to new
idea.

Thank you for reading.


Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Feb 7, 2019 at 3:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> WAL encryption will follow as an additional feature.

I don't think WAL encryption is an optional feature.  You can argue
about whether it's useful to encrypt the disk files in the first place
given that there's no privilege boundary between the OS user and the
database, but a lot of people seem to think it is and maybe they're
right.  However, who can justify encrypting only SOME of the disk
files and not others?  I mean, maybe you could justify not encryption
the SLRU files on the grounds that they probably don't leak much in
the way of interesting information, but the WAL files certainly do --
your data is there, just as much as in the data files themselves.

To be honest, I think there is a lot to like about the patches
Cybertec has proposed.  Those patches don't have all of the fancy
key-management stuff that you are proposing here, but maybe that
stuff, if we want it, could be added, rather than starting over from
scratch.  It seems to me that those patches get a lot of things right.
In particular, it looked to me when I looked at them like they made a
pretty determined effort to encrypt every byte that might go down to
the disk.  It seems to me that you if you want encryption, you want
that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Mar 1, 2019 at 3:52 PM Haribabu Kommi <kommi.haribabu@gmail.com> wrote:
> The Cybertec proposed patches are doing the encryption at the instance
> level, AFAIK, the current discussion is also trying to reduce the scope of the
> encryption to object level like (tablesapce, database or table) to avoid the encryption
> performance impact for the databases, tables that don't need it.

The trick there is that it becomes difficult to figure out which keys
to use for certain things.  For example, you could say, well, this WAL
record is for a table that is encrypted with key 123, so let's use key
123 to encrypt the WAL record also.  So far, so good.  But then how do
you encrypt, say, a logical decoding spill file?  That could have data
in it mixed together from multiple relations, IIUC.  Or what do you do
about SLRUs or other global structures?  If you just exclude that
stuff from the scope of encryption, then you aren't helping the people
who want to Just Encrypt Everything.

Now that having been said I bet a lot of people would find it pretty
cool if we could make this work on a per-table basis.  And I'm not
opposed to that.  I just think it's really hard.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Mar 2, 2019 at 5:27 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Feb 7, 2019 at 3:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > WAL encryption will follow as an additional feature.
>
> I don't think WAL encryption is an optional feature.  You can argue
> about whether it's useful to encrypt the disk files in the first place
> given that there's no privilege boundary between the OS user and the
> database, but a lot of people seem to think it is and maybe they're
> right.  However, who can justify encrypting only SOME of the disk
> files and not others?  I mean, maybe you could justify not encryption
> the SLRU files on the grounds that they probably don't leak much in
> the way of interesting information, but the WAL files certainly do --
> your data is there, just as much as in the data files themselves.
>

Agreed.

> To be honest, I think there is a lot to like about the patches
> Cybertec has proposed.  Those patches don't have all of the fancy
> key-management stuff that you are proposing here, but maybe that
> stuff, if we want it, could be added, rather than starting over from
> scratch.  It seems to me that those patches get a lot of things right.
> In particular, it looked to me when I looked at them like they made a
> pretty determined effort to encrypt every byte that might go down to
> the disk.  It seems to me that you if you want encryption, you want
> that.
>

Agreed. I think the patch lacks the key management stuff: 2-tier key
architecture and integration of postgres with key management systems.
I'd like to work together and can propose the patch of key management
stuff to the proposed patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Sat, Mar 2, 2019 at 6:23 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Mar 1, 2019 at 3:52 PM Haribabu Kommi <kommi.haribabu@gmail.com> wrote:
> > The Cybertec proposed patches are doing the encryption at the instance
> > level, AFAIK, the current discussion is also trying to reduce the scope of the
> > encryption to object level like (tablesapce, database or table) to avoid the encryption
> > performance impact for the databases, tables that don't need it.
>
> The trick there is that it becomes difficult to figure out which keys
> to use for certain things.  For example, you could say, well, this WAL
> record is for a table that is encrypted with key 123, so let's use key
> 123 to encrypt the WAL record also.  So far, so good.  But then how do
> you encrypt, say, a logical decoding spill file?  That could have data
> in it mixed together from multiple relations, IIUC.

I think that there is no need to use the same key for both the spill
files and WAL because only one process encrypt/decrypt spill files. We
can use something like temporary key for that use case, which is used
by only one process and lives during process lifetime (or transaction
lifetime). The same is true for for other temporary files such as
tuplesort and tuplestore, although maybe we need tricks for shared
tuplestore.

> Or what do you do
> about SLRUs or other global structures?  If you just exclude that
> stuff from the scope of encryption, then you aren't helping the people
> who want to Just Encrypt Everything.

Why do people want to just encrypt everything? For satisfying some
security compliance?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Masahiko Sawada wrote:
> Why do people want to just encrypt everything? For satisfying some
> security compliance?

I'd say that TDE primarily protects you from masked ninjas that
break into your server room and rip out the disks with your database
on them.

Or from people stealing your file system backups that you leave
lying around in public.

My guess is that this requirement almost always comes from security
departments that don't know a lot about the typical security threats
that databases face, or (worse) from lawmakers.

And these are probably the people who will insist that *everything*
is encrypted, even your commit log (unencrypted log? everyone can
read the commits?).

Yours,
Laurenz Albe

Or on your laptop



On 3/4/19 11:55 AM, Laurenz Albe wrote:
> Masahiko Sawada wrote:
>> Why do people want to just encrypt everything? For satisfying some
>> security compliance?
> I'd say that TDE primarily protects you from masked ninjas that
> break into your server room and rip out the disks with your database
> on them.
>
> Or from people stealing your file system backups that you leave
> lying around in public.
>
> My guess is that this requirement almost always comes from security
> departments that don't know a lot about the typical security threats
> that databases face, or (worse) from lawmakers.
>
> And these are probably the people who will insist that *everything*
> is encrypted, even your commit log (unencrypted log? everyone can
> read the commits?).
>
> Yours,
> Laurenz Albe
>
>
>
>

On 3/4/19 6:55 PM, Laurenz Albe wrote:
> Masahiko Sawada wrote:
>> Why do people want to just encrypt everything? For satisfying some
>> security compliance?
> 
> I'd say that TDE primarily protects you from masked ninjas that
> break into your server room and rip out the disks with your database
> on them.
> 
> Or from people stealing your file system backups that you leave
> lying around in public.
> 
> My guess is that this requirement almost always comes from security
> departments that don't know a lot about the typical security threats
> that databases face, or (worse) from lawmakers.
> 
> And these are probably the people who will insist that *everything*
> is encrypted, even your commit log (unencrypted log? everyone can
> read the commits?).
> 

IMHO it's a sound design principle - deny access by default, then allow
for specific cases. It's much easier to reason about, and also validate
such solutions.

It's pretty much the same reason why firewall rules generally prohibit
everything by default, and then only allow access for specific ports,
from specific IP ranges, etc. Doing it the other way around is futile.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 3/4/19 6:40 AM, Masahiko Sawada wrote:
> On Sat, Mar 2, 2019 at 5:27 AM Robert Haas <robertmhaas@gmail.com> wrote:
>>
>> On Thu, Feb 7, 2019 at 3:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> WAL encryption will follow as an additional feature.
>>
>> I don't think WAL encryption is an optional feature.  You can argue
>> about whether it's useful to encrypt the disk files in the first place
>> given that there's no privilege boundary between the OS user and the
>> database, but a lot of people seem to think it is and maybe they're
>> right.  However, who can justify encrypting only SOME of the disk
>> files and not others?  I mean, maybe you could justify not encryption
>> the SLRU files on the grounds that they probably don't leak much in
>> the way of interesting information, but the WAL files certainly do --
>> your data is there, just as much as in the data files themselves.
>>
> 
> Agreed.
> 
>> To be honest, I think there is a lot to like about the patches
>> Cybertec has proposed.  Those patches don't have all of the fancy
>> key-management stuff that you are proposing here, but maybe that
>> stuff, if we want it, could be added, rather than starting over from
>> scratch.  It seems to me that those patches get a lot of things right.
>> In particular, it looked to me when I looked at them like they made a
>> pretty determined effort to encrypt every byte that might go down to
>> the disk.  It seems to me that you if you want encryption, you want
>> that.
>>
> 
> Agreed. I think the patch lacks the key management stuff: 2-tier key
> architecture and integration of postgres with key management systems.
> I'd like to work together and can propose the patch of key management
> stuff to the proposed patch.
> 

Sounds like a plan. It'd be nice to come up with a unified version of
those two patches, combining the good pieces from both.

I wonder how other databases deal with key management? Surely we're not
the first/only database that tries to do transparent encryption, so
perhaps we could learn something from others? For example, do they use
this 2-tier key architecture? How do they do key management? etc.

I don't say we should copy from them, but it'd allow us to (a) avoid
making the same mistakes and (b) build a solution the users are already
somewhat familiar with.

May I suggest creating a page on the PostgreSQL wiki, explaining the
design and updating it as the discussion develops? It's rather difficult
to follow all the different sub-threads, and IIRC some larger patches
used that successfully for this purpose.

See for example:

* https://wiki.postgresql.org/wiki/Parallel_External_Sort
* https://wiki.postgresql.org/wiki/Parallel_Internal_Sort


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Mar 5, 2019 at 3:46 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
>
>
> On 3/4/19 6:40 AM, Masahiko Sawada wrote:
> > On Sat, Mar 2, 2019 at 5:27 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >>
> >> On Thu, Feb 7, 2019 at 3:28 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>> WAL encryption will follow as an additional feature.
> >>
> >> I don't think WAL encryption is an optional feature.  You can argue
> >> about whether it's useful to encrypt the disk files in the first place
> >> given that there's no privilege boundary between the OS user and the
> >> database, but a lot of people seem to think it is and maybe they're
> >> right.  However, who can justify encrypting only SOME of the disk
> >> files and not others?  I mean, maybe you could justify not encryption
> >> the SLRU files on the grounds that they probably don't leak much in
> >> the way of interesting information, but the WAL files certainly do --
> >> your data is there, just as much as in the data files themselves.
> >>
> >
> > Agreed.
> >
> >> To be honest, I think there is a lot to like about the patches
> >> Cybertec has proposed.  Those patches don't have all of the fancy
> >> key-management stuff that you are proposing here, but maybe that
> >> stuff, if we want it, could be added, rather than starting over from
> >> scratch.  It seems to me that those patches get a lot of things right.
> >> In particular, it looked to me when I looked at them like they made a
> >> pretty determined effort to encrypt every byte that might go down to
> >> the disk.  It seems to me that you if you want encryption, you want
> >> that.
> >>
> >
> > Agreed. I think the patch lacks the key management stuff: 2-tier key
> > architecture and integration of postgres with key management systems.
> > I'd like to work together and can propose the patch of key management
> > stuff to the proposed patch.
> >
>
> Sounds like a plan. It'd be nice to come up with a unified version of
> those two patches, combining the good pieces from both.
>
> I wonder how other databases deal with key management? Surely we're not
> the first/only database that tries to do transparent encryption, so
> perhaps we could learn something from others? For example, do they use
> this 2-tier key architecture? How do they do key management? etc.
>
> I don't say we should copy from them, but it'd allow us to (a) avoid
> making the same mistakes and (b) build a solution the users are already
> somewhat familiar with.
>
> May I suggest creating a page on the PostgreSQL wiki, explaining the
> design and updating it as the discussion develops?

Understood. I've been researching transparent encryption of other
databases and been considering the architecture. I'll write down them
to the wiki.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Mar 4, 2019 at 1:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> I think that there is no need to use the same key for both the spill
> files and WAL because only one process encrypt/decrypt spill files. We
> can use something like temporary key for that use case, which is used
> by only one process and lives during process lifetime (or transaction
> lifetime). The same is true for for other temporary files such as
> tuplesort and tuplestore, although maybe we need tricks for shared
> tuplestore.

Agreed.  For a shared tuplestore you need a key that is shared between
the processes involved, but it doesn't need to be the same as any
other key.  For anything that is accessed by only a single process,
that process can just generate any old key and, as long as it's
secure, it's fine.

For the WAL, you could potentially create a new WAL record type that
is basically an encrypted wrapper around another WAL record.  So if
table X is encrypted with key K1, then all of the WAL records for
table X are wrapped inside of an encrypted-record WAL record that is
encrypted with key K1.  That's useful for people who want fine-grained
encryption only of certain data.

But for people who want to just encrypt everything, you need to
encrypt the entire WAL stream, all SLRU data, etc. and that pretty
much all has to be one key (or sub-keys derived from that one key
somehow).

> > Or what do you do
> > about SLRUs or other global structures?  If you just exclude that
> > stuff from the scope of encryption, then you aren't helping the people
> > who want to Just Encrypt Everything.
>
> Why do people want to just encrypt everything? For satisfying some
> security compliance?

Yeah, I think so.  Perhaps an encrypted filesystem is a better way to
go, but some people want something that is built into the database
server.  The motivation seems to be mostly that they have a compliance
requirement -- either the database itself encrypts everything, or they
cannot use the software.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 3/3/19 21:40, Masahiko Sawada wrote:
> On Sat, Mar 2, 2019 at 5:27 AM Robert Haas <robertmhaas@gmail.com> wrote:
>> To be honest, I think there is a lot to like about the patches
>> Cybertec has proposed.  Those patches don't have all of the fancy
>> key-management stuff that you are proposing here, but maybe that
>> stuff, if we want it, could be added, rather than starting over from
>> scratch.  It seems to me that those patches get a lot of things right.
>> In particular, it looked to me when I looked at them like they made a
>> pretty determined effort to encrypt every byte that might go down to
>> the disk.  It seems to me that you if you want encryption, you want
>> that.
> 
> Agreed. I think the patch lacks the key management stuff: 2-tier key
> architecture and integration of postgres with key management systems.
> I'd like to work together and can propose the patch of key management
> stuff to the proposed patch.

Might it make sense to generalize a little bit to secret management? It
would be *great* if PostgreSQL could have a standard "secrets" API which
could then use plugins or extensions to provide an internal
implementation (software or hardware based) and/or plug in to an
external secret management service, whether an OSS package installed on
the box or some 3rd party service off the box.

The two obvious use cases are encryption keys (mentioned here) and
passwords for things like logical replication, FDWs, dblinks, other
extensions, etc. Aside from adding new encryption key secrets, the way
PostgreSQL handles the existing secrets it already has today leaves room
for improvement.

-Jeremy

-- 
Jeremy Schneider
Database Engineer
Amazon Web Services

On Wed, Mar  6, 2019 at 10:49:17AM -0800, Jeremy Schneider wrote:
> Might it make sense to generalize a little bit to secret management? It
> would be *great* if PostgreSQL could have a standard "secrets" API which
> could then use plugins or extensions to provide an internal
> implementation (software or hardware based) and/or plug in to an
> external secret management service, whether an OSS package installed on
> the box or some 3rd party service off the box.
> 
> The two obvious use cases are encryption keys (mentioned here) and
> passwords for things like logical replication, FDWs, dblinks, other
> extensions, etc. Aside from adding new encryption key secrets, the way
> PostgreSQL handles the existing secrets it already has today leaves room
> for improvement.

See this email for a possible implementation:

    https://www.postgresql.org/message-id/20190222035816.uozqvc4wjyag3pme@momjian.us

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Mar 6, 2019 at 6:32 PM Bruce Momjian <bruce@momjian.us> wrote:
> On Wed, Mar  6, 2019 at 10:49:17AM -0800, Jeremy Schneider wrote:
> > Might it make sense to generalize a little bit to secret management? It
> > would be *great* if PostgreSQL could have a standard "secrets" API which
> > could then use plugins or extensions to provide an internal
> > implementation (software or hardware based) and/or plug in to an
> > external secret management service, whether an OSS package installed on
> > the box or some 3rd party service off the box.
> >
> > The two obvious use cases are encryption keys (mentioned here) and
> > passwords for things like logical replication, FDWs, dblinks, other
> > extensions, etc. Aside from adding new encryption key secrets, the way
> > PostgreSQL handles the existing secrets it already has today leaves room
> > for improvement.
>
> See this email for a possible implementation:
>
>         https://www.postgresql.org/message-id/20190222035816.uozqvc4wjyag3pme@momjian.us

I don't think that actually does what would be needed here.
pgcryptokey can manage the keys themselves, but the secrets (i.e.
passwords) that are used to access those keys are and must be revealed
to everyone who uses them.  I think we can imagine a
secrets-management solution where that's not the case -- where you can
access an encrypted database cluster or an encrypted table or an
encrypted column or an FDW on another server without being able to
access either the encryption key or the password for that key.

Generally, I think our interest should be less in how secrets are
stored inside the database than in how we can integrate with an
external secrets-management solution, and I think that's what Jeremy
is talking about here.  I don't know exactly how that would work, but
you can imagine having a way to tell an FDW "hey, there's a password
for this server, but it's not stored here -- instead go fetch secret
d41d8cd98f00b204e9800998ecf8427e" and the server does that and uses
that password for the connection.  But we don't need to solve the FDW
problem for this effort to move forward.  We do, however, need a
solution that's good enough for whatever we want to do in terms of
TDE.

If we imagine whole-database TDE, then there's really only one secret,
so there's not much to design.  We can just have a command that is
configured via a GUC that has to return the secret, and a user can put
whatever script they like in there.  But if we want to have
fine-grained TDE where different bits are encrypted with different
keys, then we have to have a way to request whichever key is needed
for a certain bit of data.  I don't know whether it's good enough to
just run a script and pass it some identifier and let it return the
corresponding key, or whether we should try to do something more
ambitious than that in the hopes of meeting more use case.  Sometimes
the perfect can be the enemy of the good, but half-baked solutions are
no good either.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Generally, I think our interest should be less in how secrets are
stored inside the database than in how we can integrate with an
external secrets-management solution, and I think that's what Jeremy
is talking about here.  I don't know exactly how that would work, but
you can imagine having a way to tell an FDW "hey, there's a password
for this server, but it's not stored here -- instead go fetch secret
d41d8cd98f00b204e9800998ecf8427e" and the server does that and uses
that password for the connection.  But we don't need to solve the FDW
problem for this effort to move forward.  We do, however, need a
solution that's good enough for whatever we want to do in terms of
TDE.

-- 
Jeremy Schneider
Database Engineer
Amazon Web Services

On Wed, Mar 6, 2019 at 12:09 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Mar 4, 2019 at 1:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > I think that there is no need to use the same key for both the spill
> > files and WAL because only one process encrypt/decrypt spill files. We
> > can use something like temporary key for that use case, which is used
> > by only one process and lives during process lifetime (or transaction
> > lifetime). The same is true for for other temporary files such as
> > tuplesort and tuplestore, although maybe we need tricks for shared
> > tuplestore.
>
> Agreed.  For a shared tuplestore you need a key that is shared between
> the processes involved, but it doesn't need to be the same as any
> other key.  For anything that is accessed by only a single process,
> that process can just generate any old key and, as long as it's
> secure, it's fine.

Thank you for the advice. Understood.

>
> For the WAL, you could potentially create a new WAL record type that
> is basically an encrypted wrapper around another WAL record.  So if
> table X is encrypted with key K1, then all of the WAL records for
> table X are wrapped inside of an encrypted-record WAL record that is
> encrypted with key K1.  That's useful for people who want fine-grained
> encryption only of certain data.
>
> But for people who want to just encrypt everything, you need to
> encrypt the entire WAL stream, all SLRU data, etc. and that pretty
> much all has to be one key (or sub-keys derived from that one key
> somehow).

Agreed.

For the WAL encryption, I wonder if we can have a encryption key
dedicated for WAL. Regardless of keys of tables and indexes all WAL
are encrypted with the WAL key. During the recovery the startup
process decrypts WAL and applies it, and then the table data will be
encrypted with its table key when flushing. So we just control the
scope of encryption object: WAL of tables and indexes etc or
everything.

>
> > > Or what do you do
> > > about SLRUs or other global structures?  If you just exclude that
> > > stuff from the scope of encryption, then you aren't helping the people
> > > who want to Just Encrypt Everything.
> >
> > Why do people want to just encrypt everything? For satisfying some
> > security compliance?
>
> Yeah, I think so.  Perhaps an encrypted filesystem is a better way to
> go, but some people want something that is built into the database
> server.  The motivation seems to be mostly that they have a compliance
> requirement -- either the database itself encrypts everything, or they
> cannot use the software.
>

Understood. Maybe we need a option to control encrypting database
including all meta data or excluding them.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> On Wed, Mar 6, 2019 at 12:09 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Mon, Mar 4, 2019 at 1:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > I think that there is no need to use the same key for both the spill
> > > files and WAL because only one process encrypt/decrypt spill files. We
> > > can use something like temporary key for that use case, which is used
> > > by only one process and lives during process lifetime (or transaction
> > > lifetime). The same is true for for other temporary files such as
> > > tuplesort and tuplestore, although maybe we need tricks for shared
> > > tuplestore.
> >
> > Agreed.  For a shared tuplestore you need a key that is shared between
> > the processes involved, but it doesn't need to be the same as any
> > other key.  For anything that is accessed by only a single process,
> > that process can just generate any old key and, as long as it's
> > secure, it's fine.
>
> Thank you for the advice. Understood.
>
> >
> > For the WAL, you could potentially create a new WAL record type that
> > is basically an encrypted wrapper around another WAL record.  So if
> > table X is encrypted with key K1, then all of the WAL records for
> > table X are wrapped inside of an encrypted-record WAL record that is
> > encrypted with key K1.  That's useful for people who want fine-grained
> > encryption only of certain data.
> >
> > But for people who want to just encrypt everything, you need to
> > encrypt the entire WAL stream, all SLRU data, etc. and that pretty
> > much all has to be one key (or sub-keys derived from that one key
> > somehow).
>
> Agreed.
>
> For the WAL encryption, I wonder if we can have a encryption key
> dedicated for WAL. Regardless of keys of tables and indexes all WAL
> are encrypted with the WAL key. During the recovery the startup
> process decrypts WAL and applies it, and then the table data will be
> encrypted with its table key when flushing. So we just control the
> scope of encryption object: WAL of tables and indexes etc or
> everything.

My point of view is that different key usually means different user. The user
who can decrypt WAL can effectively see all the data, even though another user
put them (encrypted with another key) into tables. So in this case, different
keys don't really separate users in terms of data access.

--
Antonin Houska
https://www.cybertec-postgresql.com

Antonin Houska <ah@cybertec.at> wrote:

> Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Agreed.
> >
> > For the WAL encryption, I wonder if we can have a encryption key
> > dedicated for WAL. Regardless of keys of tables and indexes all WAL
> > are encrypted with the WAL key. During the recovery the startup
> > process decrypts WAL and applies it, and then the table data will be
> > encrypted with its table key when flushing. So we just control the
> > scope of encryption object: WAL of tables and indexes etc or
> > everything.
>
> My point of view is that different key usually means different user. The user
> who can decrypt WAL can effectively see all the data, even though another user
> put them (encrypted with another key) into tables. So in this case, different
> keys don't really separate users in terms of data access.

Please ignore what I said here. You probably meant that the WAL is both
encrypted and decrypted using the same (dedicated) key.

--
Antonin Houska
https://www.cybertec-postgresql.com

On 3/8/19 5:38 PM, Antonin Houska wrote:
> Antonin Houska <ah@cybertec.at> wrote:
> 
>> Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>>> Agreed.
>>>
>>> For the WAL encryption, I wonder if we can have a encryption key
>>> dedicated for WAL. Regardless of keys of tables and indexes all WAL
>>> are encrypted with the WAL key. During the recovery the startup
>>> process decrypts WAL and applies it, and then the table data will be
>>> encrypted with its table key when flushing. So we just control the
>>> scope of encryption object: WAL of tables and indexes etc or
>>> everything.
>>
>> My point of view is that different key usually means different user. The user
>> who can decrypt WAL can effectively see all the data, even though another user
>> put them (encrypted with another key) into tables. So in this case, different
>> keys don't really separate users in terms of data access.
> 
> Please ignore what I said here. You probably meant that the WAL is both
> encrypted and decrypted using the same (dedicated) key.
> 

I think this very much depends on the threat model. If the encryption is
supposed to serve as a second access control layer (orthogonal to the
ACL stuff we already have), then a single WAL key may not be sufficient.

I may be misunderstanding the whole scheme, but it seems to me features
like logical decoding do require knowledge of the WAL key. So sessions
performing logical decoding (which are regular user sessions) would know
the WAL key, which gives them the ability to decode everything.

So if the threat model includes insider thread (someone with access to a
subset of data, gaining unauthorized access to everything), then this
would be an issue. Such bad actor might obtain access to WAL archive, or
possibly just copy the WAL segments on his own ...

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Mar 9, 2019 at 3:08 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On 3/8/19 5:38 PM, Antonin Houska wrote:
> > Antonin Houska <ah@cybertec.at> wrote:
> >
> >> Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>
> >>> Agreed.
> >>>
> >>> For the WAL encryption, I wonder if we can have a encryption key
> >>> dedicated for WAL. Regardless of keys of tables and indexes all WAL
> >>> are encrypted with the WAL key. During the recovery the startup
> >>> process decrypts WAL and applies it, and then the table data will be
> >>> encrypted with its table key when flushing. So we just control the
> >>> scope of encryption object: WAL of tables and indexes etc or
> >>> everything.
> >>
> >> My point of view is that different key usually means different user. The user
> >> who can decrypt WAL can effectively see all the data, even though another user
> >> put them (encrypted with another key) into tables. So in this case, different
> >> keys don't really separate users in terms of data access.
> >
> > Please ignore what I said here. You probably meant that the WAL is both
> > encrypted and decrypted using the same (dedicated) key.
> >
>
> I think this very much depends on the threat model. If the encryption is
> supposed to serve as a second access control layer (orthogonal to the
> ACL stuff we already have), then a single WAL key may not be sufficient.
>

Agreed.

> I may be misunderstanding the whole scheme, but it seems to me features
> like logical decoding do require knowledge of the WAL key. So sessions
> performing logical decoding (which are regular user sessions) would know
> the WAL key, which gives them the ability to decode everything.

Yeah, currently logical decoding requires the super user privilege and
it can decode everything if they have regardless of per table
privileges. So the session performing logical decoding will take the
WAL key and decode WAL with it, and can use the WAL key or temporary
key for spilled files.

I'm trying to implement TDE while not changing the current access
control behavior. That is, if a user has an access privilege of a
table he/she can access it as same as before and encrypts and decrypts
it transparently. I've considered a design of having two layers the
encryption and access control to access data; users might see
encrypted data if they have a access privilege but not have the
decryption privilege. But I think there would be two problems: the
access control layer will get complex and applications need to be
tolerate getting encrypted data.

>
> So if the threat model includes insider thread (someone with access to a
> subset of data, gaining unauthorized access to everything), then this
> would be an issue. Such bad actor might obtain access to WAL archive, or
> possibly just copy the WAL segments on his own ...
>

So I think there is no such insider thread problem, right?

We can think the threat simply, the access control is not changed. The
current design cannot prevent data from theft by malicious user who
have access privileges. That can be addressed by auditing or more
fine-grained access control.



Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Mar 14, 2019 at 8:30 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> So I think there is no such insider thread problem, right?

No, I think that's a really bad assumption.  If there's no insider
threat, then what exactly ARE you guarding against?  If somebody
steals the disk, you don't need PostgreSQL to do anything; just
encrypt the filesystem and call it good.  Encryption within the
database can only provide any value when someone already has some
degree of access to the system, and we want to prevent them from
getting more access.  Maybe they are legitimately allowed some access
and we want to keep them from exceeding those bounds, or maybe they
hacked something else to gain limited access and we want to keep them
from parleying that into more access.  Either way, they are to some
extent an insider.

And that is why I think your idea that it's OK to encrypt the WAL with
key A while the tables are meanwhile being encrypted with keys B1, B2,
..., Bn is unsound.  In such an environment, anybody who has key A
does not really need to bother compromising the other keys; they
basically get everything anyway.  There is therefore little point in
having the complexity of multiple keys; just use key A for everything
and be done with it.  To justify the complexity of multiple keys, they
need to be independent of each other from a security perspective; that
is, it must be that all copies of any given piece of data are
encrypted with the same key, and you can therefore get that data only
if you get that key.  And that means, I believe, that fine-grained
encryption must use the same key to encrypt the WAL for table T1 --
and the indexes and TOAST table for table T1 and the WAL for those --
that it uses to encrypt table T1 itself.

If we can't make that happen, then I really don't see any advantage in
supporting multiple keys.  Nobody steals the key to one office if they
can, for the same effort, steal the master key for the whole building.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Replying to myself to resend to the list, since my previous attempt
seems to have been eaten by a grue.

On Tue, Apr 30, 2019 at 1:01 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Tue, Apr 30, 2019 at 1:38 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > It seems to me that encrypting table data in WAL with multiple keys
> > reduces damage in case where a key is theft. However user who has an
> > access privilege of reading WAL can obtain all keys encrypting table
> > data in WAL in the first place.
>
> That better not be true.  If you have a design where reading the WAL
> lets you get *any* encryption key, you have a bad design, I think.  If
> you have a design where reading the WAL lets you get *every*
> encryption key, that's truly terrible.  That's strictly worse than
> full-disk encryption, which at least protects against the disk being
> stolen.
>
> > So as long as postgres's access
> > control facility works fine it's the same as having single encryption
> > key dedicated for WAL.
>
> I think of postgres's access control facility works fine there is no
> need for encryption in the first place.  If we can just use REVOKE to
> block access, we should do that and forget about encryption.  The
> value of encryption only enters the picture when someone can bypass
> the database server permissions in some way, such as by reading the
> files directly.
>
> > Or do you mean the case where a malicious user
> > steals both WAL and key A? It completely depends on from what threat
> > we want to protect data by transparent data encryption but I think we
> > should rather protect from such threat by the access control in that
> > situation. I personally don't think the having an encryption key
> > dedicated for WAL would increase risk much.
>
> Well, what threat are you trying to protect against?
>
> > FWIW, binary log encryption of MySQL uses different encryption key
> > from a key used for table[1]. The key is encrypted by the master key
> > for binary log encryption and is stored in each file headers.
>
> So, if you steal the master key for binary log encryption, you can
> decrypt everything, it sounds like.
>
> > If we use the same key to encrypt the WAL for the table and indexes
> > and TOAST table for the table, what encryption key should we use for
> > temporary files for an intermediate result?
>
> For temporary files, we can just use some temporary key that is only
> stored in server memory and only for the lifetime of the session.
> Once the session ends, we don't ever need to read that data again.
>
> > And should we use each
> > different encryption keys for WAL other than table and indexes
> > resource manager?
>
> Data other than table and index data seems like it is not very
> security-sensitive.  I'm not sure we need to encrypt it at all.  If we
> do, using one key seems fine.
>
> > The advantage of having the dedicated key for WAL
> > encryption would be to make WAL encryption more simple. If we do that,
> > we can encrypt the whole WAL page by key A like the patch proposed by
> > Antonin does).
>
> Yeah.  His design is simpler because it is coarse-grained: the whole
> cluster is encrypted, so there's no need to worry about encrypting
> data differently for different tables.
>
> > Also the advantage of having multiple tablespace keys
> > would be to make postgres enable to re-encryption without rebuilt
> > database cluster.
>
> I don't understand this part, sorry.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company



-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, May 1, 2019 at 9:30 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> Replying to myself to resend to the list, since my previous attempt
> seems to have been eaten by a grue.
>
> On Tue, Apr 30, 2019 at 1:01 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Tue, Apr 30, 2019 at 1:38 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > It seems to me that encrypting table data in WAL with multiple keys
> > > reduces damage in case where a key is theft. However user who has an
> > > access privilege of reading WAL can obtain all keys encrypting table
> > > data in WAL in the first place.
> >
> > That better not be true.  If you have a design where reading the WAL
> > lets you get *any* encryption key, you have a bad design, I think.

How does the startup process decrypt WAL during recovery without
getting any encryption key if we encrypt user data in WAL by multiple
encryption keys?

> > If
> > you have a design where reading the WAL lets you get *every*
> > encryption key, that's truly terrible.  That's strictly worse than
> > full-disk encryption, which at least protects against the disk being
> > stolen.
> >
> > > So as long as postgres's access
> > > control facility works fine it's the same as having single encryption
> > > key dedicated for WAL.
> >
> > I think of postgres's access control facility works fine there is no
> > need for encryption in the first place.  If we can just use REVOKE to
> > block access, we should do that and forget about encryption.  The
> > value of encryption only enters the picture when someone can bypass
> > the database server permissions in some way, such as by reading the
> > files directly.

Yes, I agreed.

> >
> > > Or do you mean the case where a malicious user
> > > steals both WAL and key A? It completely depends on from what threat
> > > we want to protect data by transparent data encryption but I think we
> > > should rather protect from such threat by the access control in that
> > > situation. I personally don't think the having an encryption key
> > > dedicated for WAL would increase risk much.
> >
> > Well, what threat are you trying to protect against?

Data theft bypassing PostgreSQL's ACL, for example a malicious user
thefts storage devices and reads datbase files directly.

I'm thinking that only users who have an access privilege of the
database object can get encryption key for the object. Therefore, when
a malicious user stole an encryption key by breaking the access
control system if we suppose data at rest encryption to serve as a yet
another access control layer we have to use the same encryption key
for WAL as that we used for database file. But I thought that we
should rather protect data from that situation by access control
system and managing encryption keys more robustly.

> >
> > > FWIW, binary log encryption of MySQL uses different encryption key
> > > from a key used for table[1]. The key is encrypted by the master key
> > > for binary log encryption and is stored in each file headers.
> >
> > So, if you steal the master key for binary log encryption, you can
> > decrypt everything, it sounds like.

Yes, I think so.

> >
> > > If we use the same key to encrypt the WAL for the table and indexes
> > > and TOAST table for the table, what encryption key should we use for
> > > temporary files for an intermediate result?
> >
> > For temporary files, we can just use some temporary key that is only
> > stored in server memory and only for the lifetime of the session.
> > Once the session ends, we don't ever need to read that data again.
> >

Agreed.

> > > And should we use each
> > > different encryption keys for WAL other than table and indexes
> > > resource manager?
> >
> > Data other than table and index data seems like it is not very
> > security-sensitive.  I'm not sure we need to encrypt it at all.  If we
> > do, using one key seems fine.

Agreed. But it seems not to satisfy some user who require to encrypt
everything, which we discussed before.

> >
> > > The advantage of having the dedicated key for WAL
> > > encryption would be to make WAL encryption more simple. If we do that,
> > > we can encrypt the whole WAL page by key A like the patch proposed by
> > > Antonin does).
> >
> > Yeah.  His design is simpler because it is coarse-grained: the whole
> > cluster is encrypted, so there's no need to worry about encrypting
> > data differently for different tables.
> >
> > > Also the advantage of having multiple tablespace keys
> > > would be to make postgres enable to re-encryption without rebuilt
> > > database cluster.
> >
> > I don't understand this part, sorry.

I wanted to say that if we encrypt whole database cluster by single
encryption key we would need to rebuilt the database cluster when
re-encrypt data. But if we encrypt data in tablespaces by per
tablespace encryption keys we can re-encrypt data by moving
tablespaces, without rebuilt it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, May 7, 2019 at 2:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > That better not be true.  If you have a design where reading the WAL
> > > lets you get *any* encryption key, you have a bad design, I think.
>
> How does the startup process decrypt WAL during recovery without
> getting any encryption key if we encrypt user data in WAL by multiple
> encryption keys?

The keys have to be supplied from someplace outside of the database
system.  I am imagining a command that gets run with the key ID as an
argument and is expected to print the key out on standard output for
the server to read.

I am not an encryption expert, but it's hard for me to imagine this
working any other way.  I mean, if you store the keys that you need
for decryption inside the database, isn't that the same as storing
your house key in your house, or your car key in your car?  If you
store your car key in the car, then either the car is locked from the
outside, and the key is useless to you, or the car is unlocked from
the outside, and the key is just as available to a thief as it is to
you.  Either way, it provides no security.  What you do is keep your
car key in your pocket or purse; if you try to start the car, it
"requests" the key from you as proof that you are entitled to start
it.  I think the database has to work similarly, except that rather
than protecting the act of "starting" the database, each key is
requested the first time it's needed, when it's discovered that we
need to decrypt some data encrypted with that key.

> > > Well, what threat are you trying to protect against?
>
> Data theft bypassing PostgreSQL's ACL, for example a malicious user
> thefts storage devices and reads datbase files directly.
>
> I'm thinking that only users who have an access privilege of the
> database object can get encryption key for the object. Therefore, when
> a malicious user stole an encryption key by breaking the access
> control system if we suppose data at rest encryption to serve as a yet
> another access control layer we have to use the same encryption key
> for WAL as that we used for database file. But I thought that we
> should rather protect data from that situation by access control
> system and managing encryption keys more robustly.

I don't really follow that logic.  If the encryption keys are managed
robustly enough that they cannot be stolen, then we only need one.  If
there is still enough risk of key theft that we care to protect
against it, we can't use a dedicated key for the WAL without
increasing the risk.

> > > > FWIW, binary log encryption of MySQL uses different encryption key
> > > > from a key used for table[1]. The key is encrypted by the master key
> > > > for binary log encryption and is stored in each file headers.
> > >
> > > So, if you steal the master key for binary log encryption, you can
> > > decrypt everything, it sounds like.
>
> Yes, I think so.

I am not keen to copy that design.  It sounds like having multiple
keys in this design adds a lot of complexity without adding much
security.

> > > Data other than table and index data seems like it is not very
> > > security-sensitive.  I'm not sure we need to encrypt it at all.  If we
> > > do, using one key seems fine.
>
> Agreed. But it seems not to satisfy some user who require to encrypt
> everything, which we discussed before.

Agreed.  I'm thinking possibly we need two different facilities.
Facility #1 could be whole-database encryption: everything is
encrypted with one key on a block level.  And facility #2 could be
per-table encryption: blocks for specific tables (and the related
TOAST tables, indexes, and relation forks) are encrypted with specific
keys and, in addition, the WAL records for those tables (and the
related TOAST tables, indexes, and relation forks) are encrypted with
the same key, but on a per-WAL-record level; the original WAL record
would get "wrapped" by a new WAL record that just says "I am an
encrypted WAL record, key ID %d, encrypted contents: %s" and you have
to get the key to decrypt the contents and decrypt the real WAL record
inside of it.  Then you process that interior record as normal.

I guess if you had both things, you'd want tables for which facility
#2 was enabled to bypass facility #1, so that no relation data blocks
were doubly-encrypted, to avoid the overhead.  But a WAL record would
be doubly-encrypted when both facilities are in use: the record would
get encrypted with the per-table key, and then the blocks it got
stored into would be encrypted with the cluster-wide key.

> I wanted to say that if we encrypt whole database cluster by single
> encryption key we would need to rebuilt the database cluster when
> re-encrypt data. But if we encrypt data in tablespaces by per
> tablespace encryption keys we can re-encrypt data by moving
> tablespaces, without rebuilt it.

Interesting.  I suppose that would also be true of per-table keys.
CREATE TABLE newthunk ENCRYPT WITH 'hoge' AS SELECT * FROM thunk; or
something of that sort.

Is there any real advantage of making this per-tablespace rather than
per-table in PostgreSQL's architecture? In some other systems, all the
stuff in a tablespace is glommed together into a big file or a raw
disk partiton or something, so if you used different keys for
different things in the tablespace then it might be hard to know which
key to use for which blocks, but we've got separate files for each
relation anyway. Now, that doesn't answer the question of how
recovery, which can't do pg_class lookups, knows which key to use for
which relation, but recovery can't do pg_tablespace lookups either.
But I think there's a simple answer for that: the encrypted 'wrapper'
WAL record must say which key should be used to decrypt the WAL record
inside of it.  And that must be the same key ID that should be used
for the corresponding relation files that the WAL record touches.  So
no problem!

I mean, no problem apart from writing a huge amount of very complex code...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, May 8, 2019 at 10:32 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Tue, May 7, 2019 at 2:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > That better not be true.  If you have a design where reading the WAL
> > > > lets you get *any* encryption key, you have a bad design, I think.
> >
> > How does the startup process decrypt WAL during recovery without
> > getting any encryption key if we encrypt user data in WAL by multiple
> > encryption keys?
>
> The keys have to be supplied from someplace outside of the database
> system.  I am imagining a command that gets run with the key ID as an
> argument and is expected to print the key out on standard output for
> the server to read.
>
> I am not an encryption expert, but it's hard for me to imagine this
> working any other way.  I mean, if you store the keys that you need
> for decryption inside the database, isn't that the same as storing
> your house key in your house, or your car key in your car?  If you
> store your car key in the car, then either the car is locked from the
> outside, and the key is useless to you, or the car is unlocked from
> the outside, and the key is just as available to a thief as it is to
> you.  Either way, it provides no security.  What you do is keep your
> car key in your pocket or purse; if you try to start the car, it
> "requests" the key from you as proof that you are entitled to start
> it.

Agreed, keys for decryption must be stored outside of database.

>  I think the database has to work similarly, except that rather
> than protecting the act of "starting" the database, each key is
> requested the first time it's needed, when it's discovered that we
> need to decrypt some data encrypted with that key.
>

It could depend on the design. In 2-tier key architecture that we
proposed, since all data keys that we need for encryption of table
data are encrypted and stored inside of database, we can get the
master key at once when starting database and decrypt all data keys.

> > > > Well, what threat are you trying to protect against?
> >
> > Data theft bypassing PostgreSQL's ACL, for example a malicious user
> > thefts storage devices and reads datbase files directly.
> >
> > I'm thinking that only users who have an access privilege of the
> > database object can get encryption key for the object. Therefore, when
> > a malicious user stole an encryption key by breaking the access
> > control system if we suppose data at rest encryption to serve as a yet
> > another access control layer we have to use the same encryption key
> > for WAL as that we used for database file. But I thought that we
> > should rather protect data from that situation by access control
> > system and managing encryption keys more robustly.
>
> I don't really follow that logic.  If the encryption keys are managed
> robustly enough that they cannot be stolen, then we only need one.  If
> there is still enough risk of key theft that we care to protect
> against it, we can't use a dedicated key for the WAL without
> increasing the risk.

In 2-tier key architecture design, the key dedicated for WAL (=WAL
data key) is stored inside of database and it never go out of
database, which is also true  for data keys of tables and indexes .
The master key is per database cluster and it encrypts all data key as
well before storing them to the disk. Therefore when the master key is
stolen, a malicious user can see not only all data in WAL but also all
table data, because the all data keys are decrypted with the master
key. So I thought that the situation you're concerned is where a
malicious user can see a table data of that they don't have privilege
if they stole the master key, WAL data key and WAL but not for table
data. Is that right?

>
> > > > > FWIW, binary log encryption of MySQL uses different encryption key
> > > > > from a key used for table[1]. The key is encrypted by the master key
> > > > > for binary log encryption and is stored in each file headers.
> > > >
> > > > So, if you steal the master key for binary log encryption, you can
> > > > decrypt everything, it sounds like.
> >
> > Yes, I think so.
>
> I am not keen to copy that design.  It sounds like having multiple
> keys in this design adds a lot of complexity without adding much
> security.
>
> > > > Data other than table and index data seems like it is not very
> > > > security-sensitive.  I'm not sure we need to encrypt it at all.  If we
> > > > do, using one key seems fine.
> >
> > Agreed. But it seems not to satisfy some user who require to encrypt
> > everything, which we discussed before.
>
> Agreed.  I'm thinking possibly we need two different facilities.
> Facility #1 could be whole-database encryption: everything is
> encrypted with one key on a block level.  And facility #2 could be
> per-table encryption: blocks for specific tables (and the related
> TOAST tables, indexes, and relation forks) are encrypted with specific
> keys and, in addition, the WAL records for those tables (and the
> related TOAST tables, indexes, and relation forks) are encrypted with
> the same key, but on a per-WAL-record level; the original WAL record
> would get "wrapped" by a new WAL record that just says "I am an
> encrypted WAL record, key ID %d, encrypted contents: %s" and you have
> to get the key to decrypt the contents and decrypt the real WAL record
> inside of it.  Then you process that interior record as normal.
>
> I guess if you had both things, you'd want tables for which facility
> #2 was enabled to bypass facility #1, so that no relation data blocks
> were doubly-encrypted, to avoid the overhead.  But a WAL record would
> be doubly-encrypted when both facilities are in use: the record would
> get encrypted with the per-table key, and then the blocks it got
> stored into would be encrypted with the cluster-wide key.

#2 also must encrypt system catalogs as well as specified user tables,
and temporary files are also encrypted. So the difference between #1
and #2 is whether to encrypt all data in WAL from the perspective of
encrypted objects?  Or do you think that #1 encrypts anything other
objects or files such as large objects and backup_label? If the
difference is only WAL, #2 can cover #1 by encrypting all WAL records.

>
> > I wanted to say that if we encrypt whole database cluster by single
> > encryption key we would need to rebuilt the database cluster when
> > re-encrypt data. But if we encrypt data in tablespaces by per
> > tablespace encryption keys we can re-encrypt data by moving
> > tablespaces, without rebuilt it.
>
> Interesting.  I suppose that would also be true of per-table keys.
> CREATE TABLE newthunk ENCRYPT WITH 'hoge' AS SELECT * FROM thunk; or
> something of that sort.
>
> Is there any real advantage of making this per-tablespace rather than
> per-table in PostgreSQL's architecture? In some other systems, all the
> stuff in a tablespace is glommed together into a big file or a raw
> disk partiton or something, so if you used different keys for
> different things in the tablespace then it might be hard to know which
> key to use for which blocks, but we've got separate files for each
> relation anyway. Now, that doesn't answer the question of how
> recovery, which can't do pg_class lookups, knows which key to use for
> which relation, but recovery can't do pg_tablespace lookups either.
> But I think there's a simple answer for that: the encrypted 'wrapper'
> WAL record must say which key should be used to decrypt the WAL record
> inside of it.  And that must be the same key ID that should be used
> for the corresponding relation files that the WAL record touches.  So
> no problem!

In terms of keys, one advantage could be that we have less keys with
per-tablespace keys.

Let me briefly explain the current design I'm thinking. The design
employees 2-tier key architecture. That is, a database cluster has one
master key and per-tablespace keys which are encrypted with the master
key before storing to disk. Each tablespace keys are generated
randomly inside database when CREATE TABLESPACE. The all encrypted
tablespace keys are stored together with the master key ID to the file
(say, $PGDATA/base/pg_tblsp_keys). That way, the startup process can
easily get all tablespace keys and the master key ID before starting
recovery, and therefore can get the all decrypted tablespace keys. The
reason why it doesn't store per-tablespace keys in a column of
pg_tabelspace is that we also encrypt pg_tablespace with the
tablespace key. We could take a way to not encrypt only pg_tablespace,
however it instead would require to scan pg_tablespace before
recovery, and eventually we would need to not encrypt pg_attribute
that should be encrypted.

During the recovery I'm also thinking the idea you suggested; wrapper
WAL records have tablespace OID that is the lookup key for tablespace
key and the startup process can get the tablespace key.

Given that the above design less data keys is better. Obviously
per-tablespace keys are less than per-table keys. And even if we
employee per-tablespace keys we can allow user to specify per-table
encryption by using the same encryption key within the tablespace.

FYI one advantage of per-tablespace encryption from user perspective
would be less conversion when database migration. Using
default_tablespace parameter we need less modification of create table
DDL.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, May  8, 2019 at 09:32:08AM -0400, Robert Haas wrote:
> On Tue, May 7, 2019 at 2:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > That better not be true.  If you have a design where reading the WAL
> > > > lets you get *any* encryption key, you have a bad design, I think.
> >
> > How does the startup process decrypt WAL during recovery without
> > getting any encryption key if we encrypt user data in WAL by multiple
> > encryption keys?
> 
> The keys have to be supplied from someplace outside of the database
> system.  I am imagining a command that gets run with the key ID as an
> argument and is expected to print the key out on standard output for
> the server to read.

Agreed.

> I am not an encryption expert, but it's hard for me to imagine this
> working any other way.  I mean, if you store the keys that you need
> for decryption inside the database, isn't that the same as storing
> your house key in your house, or your car key in your car?  If you
> store your car key in the car, then either the car is locked from the
> outside, and the key is useless to you, or the car is unlocked from
> the outside, and the key is just as available to a thief as it is to
> you.  Either way, it provides no security.  What you do is keep your
> car key in your pocket or purse; if you try to start the car, it
> "requests" the key from you as proof that you are entitled to start
> it.  I think the database has to work similarly, except that rather
> than protecting the act of "starting" the database, each key is
> requested the first time it's needed, when it's discovered that we
> need to decrypt some data encrypted with that key.

Two-tier encryption usually stores the encrypted data keys in the
database, and the key access password is supplied externally. 
pgcryptokey does this:

    http://momjian.us/download/pgcryptokey/

        +------------------------+
        |                        |
        |   key access password  |
        |                        |
        |  +------------------+  |
        |  |encrypted_data_key|  |
        |  +------------------+  |
        |                        |
        +------------------------+

> > > > Well, what threat are you trying to protect against?
> >
> > Data theft bypassing PostgreSQL's ACL, for example a malicious user
> > thefts storage devices and reads database files directly.
> >
> > I'm thinking that only users who have an access privilege of the
> > database object can get encryption key for the object. Therefore, when
> > a malicious user stole an encryption key by breaking the access
> > control system if we suppose data at rest encryption to serve as a yet
> > another access control layer we have to use the same encryption key
> > for WAL as that we used for database file. But I thought that we
> > should rather protect data from that situation by access control
> > system and managing encryption keys more robustly.
> 
> I don't really follow that logic.  If the encryption keys are managed
> robustly enough that they cannot be stolen, then we only need one.  If
> there is still enough risk of key theft that we care to protect
> against it, we can't use a dedicated key for the WAL without
> increasing the risk.

You can change the key access password periodically by just reencrypting
the encrypted data keys with the new key access password.

Because you need to reencrypt all data when you change the encrypted
data key, you probably need to have at least two such keys active at a
time.  I think you need an API that allows applications to just use the
most recent key, and another API which allows you to select keys by
version number.  pgcryptokey does this by allowing specification of a
encrypted data key by name or key_id.

It might be necessary to allow decryption to try several versions of a
key to see which one decrypts the data.  While this is possible with PGP
because there is a checksum payload, it isn't possible with AES256
because the input/output sizes are the same.  Checking for a valid 8k
block format or WAL format might work.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, May  9, 2019 at 05:49:12PM +0900, Masahiko Sawada wrote:
> In terms of keys, one advantage could be that we have less keys with
> per-tablespace keys.
> 
> Let me briefly explain the current design I'm thinking. The design
> employees 2-tier key architecture. That is, a database cluster has one
> master key and per-tablespace keys which are encrypted with the master
> key before storing to disk. Each tablespace keys are generated
> randomly inside database when CREATE TABLESPACE. The all encrypted
> tablespace keys are stored together with the master key ID to the file
> (say, $PGDATA/base/pg_tblsp_keys). That way, the startup process can
> easily get all tablespace keys and the master key ID before starting
> recovery, and therefore can get the all decrypted tablespace keys. The
> reason why it doesn't store per-tablespace keys in a column of
> pg_tabelspace is that we also encrypt pg_tablespace with the
> tablespace key. We could take a way to not encrypt only pg_tablespace,
> however it instead would require to scan pg_tablespace before
> recovery, and eventually we would need to not encrypt pg_attribute
> that should be encrypted.
> 
> During the recovery I'm also thinking the idea you suggested; wrapper
> WAL records have tablespace OID that is the lookup key for tablespace
> key and the startup process can get the tablespace key.
> 
> Given that the above design less data keys is better. Obviously
> per-tablespace keys are less than per-table keys. And even if we
> employee per-tablespace keys we can allow user to specify per-table
> encryption by using the same encryption key within the tablespace.
> 
> FYI one advantage of per-tablespace encryption from user perspective
> would be less conversion when database migration. Using
> default_tablespace parameter we need less modification of create table
> DDL.

I think we need to step back and see what we want to do.  There are six
levels of possible encryption:

1.  client-side column encryption
2.  server-side column encryption
3.  table-level
4.  database-level
5.  tablespace-level
6.  cluster-level

1 & 2 encrypt the data in the WAL automatically, and option 6 is
encrypting the entire WAL.  This leaves 3-5 as cases where there will be
mismatch between the object-level encryption and WAL.  I don't think it
is very valuable to use these options so reencryption will be easier. 
In many cases, taking any object offline might cause the application to
fail, and having multiple encrypted data keys active will allow key
replacement to be done on an as-needed basis.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Hi Masahiko,

> Let me briefly explain the current design I'm thinking. The design employees 2-tier key architecture. That is, a
databasecluster has one
 
> master key and per-tablespace keys which are encrypted with the master key before storing to disk. Each tablespace
keysare generated
 
> randomly inside database when CREATE TABLESPACE. The all encrypted tablespace keys are stored together with the
masterkey ID to the
 
> file (say, $PGDATA/base/pg_tblsp_keys). That way, the startup process can easily get all tablespace keys and the
masterkey ID before
 
> starting recovery, and therefore can get the all decrypted tablespace keys.

Your design idea sounds very similar to the current Fujitsu Enterprise Postgres (FEP) implementation of TDE.

FEP uses a master encryption key (MEK) for the database cluster. This MEK is stored in a file at some GUC variable
location.This file is encrypted using a “passphrase” known only to the administrator.
 

There are also per-tablespace keys, which are randomly generated at the time of CREATE TABLESPACE and stored in files.
Thereis one tablespace key file per tablespace. These tablespace key files are encrypted by the MEK and stored at the
locationspecified by CREATE TABLESPACE.
 

Not all tablespaces use TDE. An FEP extension of the CREATE TABLESPACE syntax, creates the tablespace key file only
whenencryption was requested.
 
e.g. CREATE TABLESPACE my_secure_tablespace LOCATION '/home/postgre/FEP/TESTING/tablespacedir' WITH
(tablespace_encryption_algorithm= 'AES256');
 

The MEK is not currently got from a third party. It is randomly generated when the master key file is first created by
anotheradded function.
 
e.g. select pgx_set_master_key('passphrase');

Kind Regards,
Peter Smith
Fujitsu Australia
Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is
commercial-in-confidenceand is intended only for the use of the above named addressee. If you are not the intended
recipient,you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you
havereceived this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000
orby reply e-mail to the sender and delete the document and all copies thereof.
 


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email
communication,it is the receiver’s responsibility to scan all communication and any files attached for computer viruses
andother defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage
(whetherdirect, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may
resultdirectly or indirectly from this communication or any files attached.
 


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty
Ltd,please email unsubscribe@fast.au.fujitsu.com
 

On Mon, May 13, 2019 at 2:09 PM Smith, Peter <peters@fast.au.fujitsu.com> wrote:
>
> Hi Masahiko,
>
> > Let me briefly explain the current design I'm thinking. The design employees 2-tier key architecture. That is, a
databasecluster has one 
> > master key and per-tablespace keys which are encrypted with the master key before storing to disk. Each tablespace
keysare generated 
> > randomly inside database when CREATE TABLESPACE. The all encrypted tablespace keys are stored together with the
masterkey ID to the 
> > file (say, $PGDATA/base/pg_tblsp_keys). That way, the startup process can easily get all tablespace keys and the
masterkey ID before 
> > starting recovery, and therefore can get the all decrypted tablespace keys.
>
> Your design idea sounds very similar to the current Fujitsu Enterprise Postgres (FEP) implementation of TDE.
>

Yeah, I studied the design of TDE from FEP as well as other database
supporting TDE.

> FEP uses a master encryption key (MEK) for the database cluster. This MEK is stored in a file at some GUC variable
location.This file is encrypted using a “passphrase” known only to the administrator. 
>
> There are also per-tablespace keys, which are randomly generated at the time of CREATE TABLESPACE and stored in
files.There is one tablespace key file per tablespace. These tablespace key files are encrypted by the MEK and stored
atthe location specified by CREATE TABLESPACE. 
>
> Not all tablespaces use TDE. An FEP extension of the CREATE TABLESPACE syntax, creates the tablespace key file only
whenencryption was requested. 
> e.g. CREATE TABLESPACE my_secure_tablespace LOCATION '/home/postgre/FEP/TESTING/tablespacedir' WITH
(tablespace_encryption_algorithm= 'AES256'); 
>
> The MEK is not currently got from a third party. It is randomly generated when the master key file is first created
byanother added function. 
> e.g. select pgx_set_master_key('passphrase');

Thank you for explaining!

I think that the main difference between FEP and our proposal is the
master key management. In our proposal postgres can get the master key
from the external key management server or services such as AWS KMS,
Gemalto KeySecure and an encrypted file by using the corresponding
plugin. We believe that this extensible architecture would be useful
for applying postgres to various systems.


Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, May 10, 2019 at 2:42 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, May  9, 2019 at 05:49:12PM +0900, Masahiko Sawada wrote:
> > In terms of keys, one advantage could be that we have less keys with
> > per-tablespace keys.
> >
> > Let me briefly explain the current design I'm thinking. The design
> > employees 2-tier key architecture. That is, a database cluster has one
> > master key and per-tablespace keys which are encrypted with the master
> > key before storing to disk. Each tablespace keys are generated
> > randomly inside database when CREATE TABLESPACE. The all encrypted
> > tablespace keys are stored together with the master key ID to the file
> > (say, $PGDATA/base/pg_tblsp_keys). That way, the startup process can
> > easily get all tablespace keys and the master key ID before starting
> > recovery, and therefore can get the all decrypted tablespace keys. The
> > reason why it doesn't store per-tablespace keys in a column of
> > pg_tabelspace is that we also encrypt pg_tablespace with the
> > tablespace key. We could take a way to not encrypt only pg_tablespace,
> > however it instead would require to scan pg_tablespace before
> > recovery, and eventually we would need to not encrypt pg_attribute
> > that should be encrypted.
> >
> > During the recovery I'm also thinking the idea you suggested; wrapper
> > WAL records have tablespace OID that is the lookup key for tablespace
> > key and the startup process can get the tablespace key.
> >
> > Given that the above design less data keys is better. Obviously
> > per-tablespace keys are less than per-table keys. And even if we
> > employee per-tablespace keys we can allow user to specify per-table
> > encryption by using the same encryption key within the tablespace.
> >
> > FYI one advantage of per-tablespace encryption from user perspective
> > would be less conversion when database migration. Using
> > default_tablespace parameter we need less modification of create table
> > DDL.
>
> I think we need to step back and see what we want to do.  There are six
> levels of possible encryption:
>
> 1.  client-side column encryption
> 2.  server-side column encryption
> 3.  table-level
> 4.  database-level
> 5.  tablespace-level
> 6.  cluster-level
>
> 1 & 2 encrypt the data in the WAL automatically, and option 6 is
> encrypting the entire WAL.  This leaves 3-5 as cases where there will be
> mismatch between the object-level encryption and WAL.  I don't think it
> is very valuable to use these options so reencryption will be easier.
> In many cases, taking any object offline might cause the application to
> fail, and having multiple encrypted data keys active will allow key
> replacement to be done on an as-needed basis.
>

Summarizing the design discussion so far and the discussion I had at
PGCon, there are several basic design items here. Each of them is
loosely related and there are trade-off.

1. Encryption Levels.
As Bruce suggested there are 6 levels.  The fine grained control will
help to suppress performance overheads of tables that we don't
actually need to encrypt. Even in terms of security it might help
since we don't give the key users who don't or cannot access to
encrypted tables. But whichever we choose the level, we can protect
data from attack bypassing PostgresSQL's ACL such as reading database
file directly, as long as we encrypt data inside database. Threats we
want to protect by has already gotten consensus so far, I think.

Among these levels, the tablespace level would be somewhat different
from others because it corresponds to physical directories rather than
database objects. So in principles it's possible that tables are
created on an encrypted tablespace while indexes are created on
non-encrypted tablespace, which does not make sense though. But having
less encryption keys would be better for simple architecture.

2. Encryption Objects.
Indexes, WAL and TOAST table pertaining to encrypted tables, and
temporary files must also be encrypted but we need to discuss whether
we encrypt non-user data as well such as SLRU data, vm and fsm, and
perhaps even other files such as 2PC state files, backend_label etc.
Encryption everything is required by some use case but it's also true
that there are users who wish to encrypt database while minimizing
performance overheads.

3. Encryption keys.
Encryption levels would be relevant with the number of encryption keys
we use. The database cluster levels would use single encryption key
and can encrypt everything easier including non-user data such as xact
WALs and SRLU data with the same key. On the other hand, for instance
the table level would use multiple keys and can encrypt tables with
different encryption keys. One advantage of having multiple keys in
database would be that it can re-encrypt encrypted database object
as-needed basis. For instance in multi tenant architecture, the
stopping database cluster would affect all services but we can
re-encrypt data one by one while minimizing downtime of each services
if we use multiple keys. Even in terms of security, having multiple
keys helps the diversification of risk.

4. Key rotation and 2 tier key hierarchy.
Another design point would be key rotation and using 2 tier key
hierarchy. Periodic key rotation is very important and is required by
some security standard.

For instance PCI DSS 3.6.4 states "Cryptographic key changes for keys
that have reached the end of their cryptoperiod (for example, after a
defined period of time has passed and/or after a certain amount of
cipher-text has been produced by a given key), as defined by the
associated application vendor or key owner, and based on industry best
practices and guidelines"[1]. A cryptoperiod is the time span during
which a specific cryptographic key is authorized for use[2] (sometimes
represents the number of transactions). It is defined based on
multiple factors such as key length, key strength, algorithms. So it
depends on users systems, key rotation can be required even every a
few months.

2 tier key hierarchy is a technique for faster key rotation; it uses 2
types of keys: master key and data key. The master key is stored
outside database whereas the data key is stored inside database. The
data key is used to encrypt actual database data and is encrypted with
the master key before storing to the disk. And the master key must be
stored outside database. When key rotation, we rotate the master key
and re-encrypt only data key rather than database data. Therefore
since we don't need to access and modify database data the key
rotation will complete in a second. Without it, we will end up with
re-encrypting database data, which could take a long time depending on
the encryption levels.

Because the data key must not be go outside database we might need to
provide a mechanism of key rotation in database side, for example
providing pg_rotate_encryption_key() SQL function or ALTER SYSTEM
ROTATE KEY SQL command so that PostgreSQL rotate data keys with the
new master key  So it might also require PostgreSQL to get the new
master key from an external location within the command, and therefore
it's relevant with key management.

5. Key management.
Encryption key (the master key in 2 tier key hierarchy) must be taken
from an external location. If only getting, it might be enough to get
it using shell command like curl. However it's more secure if we can
seamlessly integrate PostgreSQL with key management services. And DBAs
would no longer need to care key lifecycles. In addition to that,
dedicated programs or knowledge will not be necessary in individual
user systems. Furthermore, integration with KMS might be helpful even
for column-level encryption; we could combine pgcrypto with it to
provide per column TDE.

Apart from the above discussion, there are random concerns about the
design regarding to the fine grained design. For WAL encryption, as a
result of discussion so far I'm going to use the same encryption for
WAL encryption as that used for tables. Given that approach, it would
be required to make utility commands that read WAL (pg_waldump and
pg_rewind) be able to get arbitrary encryption keys. pg_waldump might
require even an encryption keys of WAL of which table has already been
dropped. As I discussed at PGCon[3], by rearranging WAL format would
solve this issue but it doesn't resolve fundamental issue.

Also, for system catalog encryption, it could be a hard part. System
catalogs are initially created at initdb time and created by copying
from template1 when CREATE DATABASE. Therefore we would need to either
modify initdb so that it's aware of encryption keys and KMS or modify
database creation so that it copies database file while encrypting
them.

There are two proposals so far: the cluster wide encryption and per
tablespace encryption. It's good if we pick a good thing from each
proposals in order to provide an useful TDE feature to users. And
whatever design the community accepts I'd like to contribute to do
that.

Comments and feedback are very welcome!

[1] https://www.pcisecuritystandards.org/document_library?category=pcidss&document=pci_dss
[2] https://en.wikipedia.org/wiki/Cryptoperiod
[3] https://www.slideshare.net/masahikosawada98/transparent-data-encryption-in-postgresql/28


Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Jun  5, 2019 at 11:54:04AM +0900, Masahiko Sawada wrote:
> On Fri, May 10, 2019 at 2:42 AM Bruce Momjian <bruce@momjian.us> wrote:
> > I think we need to step back and see what we want to do.  There are six
> > levels of possible encryption:
> >
> > 1.  client-side column encryption
> > 2.  server-side column encryption
> > 3.  table-level
> > 4.  database-level
> > 5.  tablespace-level
> > 6.  cluster-level
> >
> > 1 & 2 encrypt the data in the WAL automatically, and option 6 is
> > encrypting the entire WAL.  This leaves 3-5 as cases where there will be
> > mismatch between the object-level encryption and WAL.  I don't think it
> > is very valuable to use these options so reencryption will be easier.
> > In many cases, taking any object offline might cause the application to
> > fail, and having multiple encrypted data keys active will allow key
> > replacement to be done on an as-needed basis.
> >
> 
> Summarizing the design discussion so far and the discussion I had at
> PGCon, there are several basic design items here. Each of them is
> loosely related and there are trade-off.
> 
> 1. Encryption Levels.
> As Bruce suggested there are 6 levels.  The fine grained control will
> help to suppress performance overheads of tables that we don't
> actually need to encrypt. Even in terms of security it might help
> since we don't give the key users who don't or cannot access to
> encrypted tables. But whichever we choose the level, we can protect
> data from attack bypassing PostgresSQL's ACL such as reading database
> file directly, as long as we encrypt data inside database. Threats we
> want to protect by has already gotten consensus so far, I think.

I think level 6 is an obvious must-have.  I think the big question is
whether we gain enough by implementing levels 3-5 compared to the
complexity of the code and user interface.  

The big question is how many people will be mixing encrypted and
unencrypted data in the same cluster, and care about performance?  Just
because someone might care is not enough of a justification.  They can
certainly create separate encrypted and non-encrypted clusters. Can we
implement level 6 and then implement levels 3-5 later if desired?

> Among these levels, the tablespace level would be somewhat different
> from others because it corresponds to physical directories rather than
> database objects. So in principles it's possible that tables are
> created on an encrypted tablespace while indexes are created on
> non-encrypted tablespace, which does not make sense though. But having
> less encryption keys would be better for simple architecture.

How would you configure the WAL to know which key to use if we did #5?
Wouldn't system tables and statistics, and perhaps referential integry
allow for information leakage?

> 2. Encryption Objects.
> Indexes, WAL and TOAST table pertaining to encrypted tables, and
> temporary files must also be encrypted but we need to discuss whether
> we encrypt non-user data as well such as SLRU data, vm and fsm, and
> perhaps even other files such as 2PC state files, backend_label etc.
> Encryption everything is required by some use case but it's also true
> that there are users who wish to encrypt database while minimizing
> performance overheads.

I don't think we need to encrypt the "status" files like SLRU data, vm
and fsm.

> 3. Encryption keys.
> Encryption levels would be relevant with the number of encryption keys
> we use. The database cluster levels would use single encryption key
> and can encrypt everything easier including non-user data such as xact
> WALs and SRLU data with the same key. On the other hand, for instance
> the table level would use multiple keys and can encrypt tables with
> different encryption keys. One advantage of having multiple keys in
> database would be that it can re-encrypt encrypted database object
> as-needed basis. For instance in multi tenant architecture, the
> stopping database cluster would affect all services but we can
> re-encrypt data one by one while minimizing downtime of each services
> if we use multiple keys. Even in terms of security, having multiple
> keys helps the diversification of risk.

I agree we need a 2 tier key hierarchy.   See my pgcryptokey extension
as an example:

    http://momjian.us/download/pgcryptokey/

> Apart from the above discussion, there are random concerns about the
> design regarding to the fine grained design. For WAL encryption, as a
> result of discussion so far I'm going to use the same encryption for
> WAL encryption as that used for tables. Given that approach, it would
> be required to make utility commands that read WAL (pg_waldump and
> pg_rewind) be able to get arbitrary encryption keys. pg_waldump might
> require even an encryption keys of WAL of which table has already been
> dropped. As I discussed at PGCon[3], by rearranging WAL format would
> solve this issue but it doesn't resolve fundamental issue.

Good point about pg_waldump.  I am a little worried we might open a
security hole making a new API so they work, so maybe we should avoid
it.

> Also, for system catalog encryption, it could be a hard part. System
> catalogs are initially created at initdb time and created by copying
> from template1 when CREATE DATABASE. Therefore we would need to either
> modify initdb so that it's aware of encryption keys and KMS or modify
> database creation so that it copies database file while encrypting
> them.

I assume initdb will use the same API that you would use to start the
server itself, e.g., type in a password, or contact a key server.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 13, 2019 at 3:48 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Wed, Jun  5, 2019 at 11:54:04AM +0900, Masahiko Sawada wrote:
> > On Fri, May 10, 2019 at 2:42 AM Bruce Momjian <bruce@momjian.us> wrote:
> > > I think we need to step back and see what we want to do.  There are six
> > > levels of possible encryption:
> > >
> > > 1.  client-side column encryption
> > > 2.  server-side column encryption
> > > 3.  table-level
> > > 4.  database-level
> > > 5.  tablespace-level
> > > 6.  cluster-level
> > >
> > > 1 & 2 encrypt the data in the WAL automatically, and option 6 is
> > > encrypting the entire WAL.  This leaves 3-5 as cases where there will be
> > > mismatch between the object-level encryption and WAL.  I don't think it
> > > is very valuable to use these options so reencryption will be easier.
> > > In many cases, taking any object offline might cause the application to
> > > fail, and having multiple encrypted data keys active will allow key
> > > replacement to be done on an as-needed basis.
> > >
> >
> > Summarizing the design discussion so far and the discussion I had at
> > PGCon, there are several basic design items here. Each of them is
> > loosely related and there are trade-off.
> >
> > 1. Encryption Levels.
> > As Bruce suggested there are 6 levels.  The fine grained control will
> > help to suppress performance overheads of tables that we don't
> > actually need to encrypt. Even in terms of security it might help
> > since we don't give the key users who don't or cannot access to
> > encrypted tables. But whichever we choose the level, we can protect
> > data from attack bypassing PostgresSQL's ACL such as reading database
> > file directly, as long as we encrypt data inside database. Threats we
> > want to protect by has already gotten consensus so far, I think.
>
> I think level 6 is an obvious must-have.  I think the big question is
> whether we gain enough by implementing levels 3-5 compared to the
> complexity of the code and user interface.
>
> The big question is how many people will be mixing encrypted and
> unencrypted data in the same cluster, and care about performance?  Just
> because someone might care is not enough of a justification.  They can
> certainly create separate encrypted and non-encrypted clusters. Can we
> implement level 6 and then implement levels 3-5 later if desired?
>

I guess most users are interested in performance. Users don't want to
sacrifice performance for security and vice versa. Fine grained
control would allow us to seek a compromise point.

From another point of view, there are our clients who want rather to
use different keys from keys other systems using for security reason
in multi tenant environment. Also, when key leakage they need to
re-encrypt database data. This feature would help such situations. We
can use different keys for each systems and can re-encrypt data
without database down.

> > Among these levels, the tablespace level would be somewhat different
> > from others because it corresponds to physical directories rather than
> > database objects. So in principles it's possible that tables are
> > created on an encrypted tablespace while indexes are created on
> > non-encrypted tablespace, which does not make sense though. But having
> > less encryption keys would be better for simple architecture.
>
> How would you configure the WAL to know which key to use if we did #5?
> Wouldn't system tables and statistics, and perhaps referential integry
> allow for information leakage?

We use a something like a map between tablespace oid and encryption
key as a separate file (maybe stored in $PGDATA/global), called
keyring. Using the keyring we can obtain encryption key by tablespace
oid. For WAL, we add a flag to XLogRecord which indicates whether the
WAL record is encrypted, and we already have relfilenode in the header
data of WAL. So we can obtain the tablespace oid from the part and
obtain the corresponding encryption key.

>
> > 2. Encryption Objects.
> > Indexes, WAL and TOAST table pertaining to encrypted tables, and
> > temporary files must also be encrypted but we need to discuss whether
> > we encrypt non-user data as well such as SLRU data, vm and fsm, and
> > perhaps even other files such as 2PC state files, backend_label etc.
> > Encryption everything is required by some use case but it's also true
> > that there are users who wish to encrypt database while minimizing
> > performance overheads.
>
> I don't think we need to encrypt the "status" files like SLRU data, vm
> and fsm.

I agree.

>
> > 3. Encryption keys.
> > Encryption levels would be relevant with the number of encryption keys
> > we use. The database cluster levels would use single encryption key
> > and can encrypt everything easier including non-user data such as xact
> > WALs and SRLU data with the same key. On the other hand, for instance
> > the table level would use multiple keys and can encrypt tables with
> > different encryption keys. One advantage of having multiple keys in
> > database would be that it can re-encrypt encrypted database object
> > as-needed basis. For instance in multi tenant architecture, the
> > stopping database cluster would affect all services but we can
> > re-encrypt data one by one while minimizing downtime of each services
> > if we use multiple keys. Even in terms of security, having multiple
> > keys helps the diversification of risk.
>
> I agree we need a 2 tier key hierarchy.   See my pgcryptokey extension
> as an example:
>
>         http://momjian.us/download/pgcryptokey/

Thanks.

>
> > Apart from the above discussion, there are random concerns about the
> > design regarding to the fine grained design. For WAL encryption, as a
> > result of discussion so far I'm going to use the same encryption for
> > WAL encryption as that used for tables. Given that approach, it would
> > be required to make utility commands that read WAL (pg_waldump and
> > pg_rewind) be able to get arbitrary encryption keys. pg_waldump might
> > require even an encryption keys of WAL of which table has already been
> > dropped. As I discussed at PGCon[3], by rearranging WAL format would
> > solve this issue but it doesn't resolve fundamental issue.
>
> Good point about pg_waldump.  I am a little worried we might open a
> security hole making a new API so they work, so maybe we should avoid
> it.

Yeah, in principle since data key of 2 tier key architecture should
not go outside database I think we should not tell data keys to
utility commands. So the rearranging WAL format seems to be a better
solution but is there any reason why the main data is placed at end of
WAL record? I wonder if we can assemble WAL records as following order
and encrypt only 3 and 4.

1. Header data (XLogRecord and other headers)
2. Main data (xl_heap_insert, xl_heap_update etc + related data)
3. Block data (Tuple data, FPI)
4. Sub data (e.g tuple data for logical decoding)

>
> > Also, for system catalog encryption, it could be a hard part. System
> > catalogs are initially created at initdb time and created by copying
> > from template1 when CREATE DATABASE. Therefore we would need to either
> > modify initdb so that it's aware of encryption keys and KMS or modify
> > database creation so that it copies database file while encrypting
> > them.
>
> I assume initdb will use the same API that you would use to start the
> server itself, e.g., type in a password, or contact a key server.

I realized that in XTS encryption mode since we craft the tweak using
relfilenode we will need to have the different tweaks for system
catalogs in new database would change. So we might need to re-encrypt
system catalogs when CREATE DATABASE after all. I suspect that even
the cluster-wide encryption has the same problem.


Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
> On Thu, Jun 13, 2019 at 3:48 AM Bruce Momjian <bruce@momjian.us> wrote:
> > The big question is how many people will be mixing encrypted and
> > unencrypted data in the same cluster, and care about performance?  Just
> > because someone might care is not enough of a justification.  They can
> > certainly create separate encrypted and non-encrypted clusters. Can we
> > implement level 6 and then implement levels 3-5 later if desired?
> 
> I guess most users are interested in performance. Users don't want to
> sacrifice performance for security and vice versa. Fine grained
> control would allow us to seek a compromise point.

Well, what does that add to the argument?  Yes, everyone cares about
performance, but it is the magnitude of the performance impact vs. the
complexity that is the issue here.  Also, by definition, users will
trade performance for security because encrypting data will slow down
the database.  The open question is how much, and if that overhead is
reasonable based on the complexity.

What I don't want to do is to design a system that is more complex than
required, and it might become so complex we might never get it done.

> > How would you configure the WAL to know which key to use if we did #5?
> > Wouldn't system tables and statistics, and perhaps referential integry
> > allow for information leakage?
> 
> We use a something like a map between tablespace oid and encryption
> key as a separate file (maybe stored in $PGDATA/global), called
> keyring. Using the keyring we can obtain encryption key by tablespace
> oid. For WAL, we add a flag to XLogRecord which indicates whether the
> WAL record is encrypted, and we already have relfilenode in the header
> data of WAL. So we can obtain the tablespace oid from the part and
> obtain the corresponding encryption key.

OK.

> > > 2. Encryption Objects.
> > > Indexes, WAL and TOAST table pertaining to encrypted tables, and
> > > temporary files must also be encrypted but we need to discuss whether
> > > we encrypt non-user data as well such as SLRU data, vm and fsm, and
> > > perhaps even other files such as 2PC state files, backend_label etc.
> > > Encryption everything is required by some use case but it's also true
> > > that there are users who wish to encrypt database while minimizing
> > > performance overheads.
> >
> > I don't think we need to encrypt the "status" files like SLRU data, vm
> > and fsm.
> 
> I agree.

Good.

> > Good point about pg_waldump.  I am a little worried we might open a
> > security hole making a new API so they work, so maybe we should avoid
> > it.
> 
> Yeah, in principle since data key of 2 tier key architecture should
> not go outside database I think we should not tell data keys to
> utility commands. So the rearranging WAL format seems to be a better
> solution but is there any reason why the main data is placed at end of
> WAL record? I wonder if we can assemble WAL records as following order
> and encrypt only 3 and 4.
> 
> 1. Header data (XLogRecord and other headers)
> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
> 3. Block data (Tuple data, FPI)
> 4. Sub data (e.g tuple data for logical decoding)

Yes, that does sound like a reasonable idea.  It is similar to us not
encrypting the clog --- there is little value.  However, if we only
encrypt the cluster, we don't need to expose the relfilenode and we can
just encrypt the entire WAL --- I like that simplicity.  We might find
that the complexity of encrypting only certain tablespaces makes the
system slower than just encrypting the entire cluster.

> > > Also, for system catalog encryption, it could be a hard part. System
> > > catalogs are initially created at initdb time and created by copying
> > > from template1 when CREATE DATABASE. Therefore we would need to either
> > > modify initdb so that it's aware of encryption keys and KMS or modify
> > > database creation so that it copies database file while encrypting
> > > them.
> >
> > I assume initdb will use the same API that you would use to start the
> > server itself, e.g., type in a password, or contact a key server.
> 
> I realized that in XTS encryption mode since we craft the tweak using
> relfilenode we will need to have the different tweaks for system
> catalogs in new database would change. So we might need to re-encrypt
> system catalogs when CREATE DATABASE after all. I suspect that even
> the cluster-wide encryption has the same problem.

Yes, this is why I want to just do cluster-wide encryption at this
stage.

In addition, while the 8k blocks would use a block cipher, the WAL would
likely use a stream cipher, and it will be very hard to use multiple
stream ciphers in a single WAL file.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 13, 2019 at 11:07:25AM -0400, Bruce Momjian wrote:
>On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
>> On Thu, Jun 13, 2019 at 3:48 AM Bruce Momjian <bruce@momjian.us> wrote:
>> > The big question is how many people will be mixing encrypted and
>> > unencrypted data in the same cluster, and care about performance?  Just
>> > because someone might care is not enough of a justification.  They can
>> > certainly create separate encrypted and non-encrypted clusters. Can we
>> > implement level 6 and then implement levels 3-5 later if desired?
>>
>> I guess most users are interested in performance. Users don't want to
>> sacrifice performance for security and vice versa. Fine grained
>> control would allow us to seek a compromise point.
>
>Well, what does that add to the argument?  Yes, everyone cares about
>performance, but it is the magnitude of the performance impact vs. the
>complexity that is the issue here.  Also, by definition, users will
>trade performance for security because encrypting data will slow down
>the database.  The open question is how much, and if that overhead is
>reasonable based on the complexity.
>
>What I don't want to do is to design a system that is more complex than
>required, and it might become so complex we might never get it done.
>

IMHO we should implement the simplest system possible, and optimize the
hell out of it without sacrificing any safety/security aspects. No smart
tunables, no extra GUCs to trade security for performance, nothing.

Then once we have this working, we can see what the impact is, and make
informed choices based on that. It's really hard to make good choices
based on speculation, which is all we have at this point. And the danger
is we'll end up with overly complex system with many parameters - which
is pretty bad when the configuration impacts security, because regular
users may not reaslise the consequences (and we'll get blamed for it).

Also, in my experience the deployments that really need this sort of
encryption tend to be quite valuable, and the owners will be happy with
higher hardware costs to compensate for the performance impact, if it
gives them the feature. So even if the performance impact is 20% (worst
case estimate), I'd say that may be acceptable.

>> > How would you configure the WAL to know which key to use if we did #5?
>> > Wouldn't system tables and statistics, and perhaps referential integry
>> > allow for information leakage?
>>
>> We use a something like a map between tablespace oid and encryption
>> key as a separate file (maybe stored in $PGDATA/global), called
>> keyring. Using the keyring we can obtain encryption key by tablespace
>> oid. For WAL, we add a flag to XLogRecord which indicates whether the
>> WAL record is encrypted, and we already have relfilenode in the header
>> data of WAL. So we can obtain the tablespace oid from the part and
>> obtain the corresponding encryption key.
>
>OK.
>
>> > > 2. Encryption Objects.
>> > > Indexes, WAL and TOAST table pertaining to encrypted tables, and
>> > > temporary files must also be encrypted but we need to discuss whether
>> > > we encrypt non-user data as well such as SLRU data, vm and fsm, and
>> > > perhaps even other files such as 2PC state files, backend_label etc.
>> > > Encryption everything is required by some use case but it's also true
>> > > that there are users who wish to encrypt database while minimizing
>> > > performance overheads.
>> >
>> > I don't think we need to encrypt the "status" files like SLRU data, vm
>> > and fsm.
>>
>> I agree.
>
>Good.
>
>> > Good point about pg_waldump.  I am a little worried we might open a
>> > security hole making a new API so they work, so maybe we should avoid
>> > it.
>>
>> Yeah, in principle since data key of 2 tier key architecture should
>> not go outside database I think we should not tell data keys to
>> utility commands. So the rearranging WAL format seems to be a better
>> solution but is there any reason why the main data is placed at end of
>> WAL record? I wonder if we can assemble WAL records as following order
>> and encrypt only 3 and 4.
>>
>> 1. Header data (XLogRecord and other headers)
>> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
>> 3. Block data (Tuple data, FPI)
>> 4. Sub data (e.g tuple data for logical decoding)
>
>Yes, that does sound like a reasonable idea.  It is similar to us not
>encrypting the clog --- there is little value.  However, if we only
>encrypt the cluster, we don't need to expose the relfilenode and we can
>just encrypt the entire WAL --- I like that simplicity.  We might find
>that the complexity of encrypting only certain tablespaces makes the
>system slower than just encrypting the entire cluster.
>

I personally find the idea of encrypting tablespaces rather strange.
Tablespaces are meant to define hysical location for objects, but this
would also use them to "mark" objects as encrypted or not. That just
seems misguided and would make the life harder for many users.

For example, what if I don't have any tablespaces (except for the
default one), but I want to encrypt only some objects? Suddenly I have
to create a tablespace, which will however cause various difficulties
down the road (during pg_basebackup, etc.).

If we really want to allow encrypting of just some objects (and I'm not
saying we need to), we should either allow defining that for individual
objects, or invent some new logical grouping of objects (where each
group is encrypted or not as a whole).

FWIW my impression is that we only really consider tablespace encryption
because Oracle has it, so users may be familiar with it. That however
ignores that Oracle actually allocates data for a tablespace (IIRC there
is a set of files per tablespace, shared by all objects in it), while
PostgreSQL has files per object.


>> > > Also, for system catalog encryption, it could be a hard part. System
>> > > catalogs are initially created at initdb time and created by copying
>> > > from template1 when CREATE DATABASE. Therefore we would need to either
>> > > modify initdb so that it's aware of encryption keys and KMS or modify
>> > > database creation so that it copies database file while encrypting
>> > > them.
>> >
>> > I assume initdb will use the same API that you would use to start the
>> > server itself, e.g., type in a password, or contact a key server.
>>
>> I realized that in XTS encryption mode since we craft the tweak using
>> relfilenode we will need to have the different tweaks for system
>> catalogs in new database would change. So we might need to re-encrypt
>> system catalogs when CREATE DATABASE after all. I suspect that even
>> the cluster-wide encryption has the same problem.
>
>Yes, this is why I want to just do cluster-wide encryption at this
>stage.
>
>In addition, while the 8k blocks would use a block cipher, the WAL would
>likely use a stream cipher, and it will be very hard to use multiple
>stream ciphers in a single WAL file.
>

Umm, why? Why would WAL necessarily use stream cipher instead of a block
cipher with a suitable mode (say, CBC or XTS)? And even if it did use
some stream cipter, why would it be hard to use multiple ciphers?

I mean, we'd probably want/need to start new streams for each WAL
segment anyway, so that tools can read and process each WAL segment
independently. So we wouldn't get very long streams anyway.



regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Fri, Jun 14, 2019 at 12:41:17AM +0200, Tomas Vondra wrote:
> On Thu, Jun 13, 2019 at 11:07:25AM -0400, Bruce Momjian wrote:
> IMHO we should implement the simplest system possible, and optimize the
> hell out of it without sacrificing any safety/security aspects. No smart
> tunables, no extra GUCs to trade security for performance, nothing.
> 
> Then once we have this working, we can see what the impact is, and make
> informed choices based on that. It's really hard to make good choices
> based on speculation, which is all we have at this point. And the danger
> is we'll end up with overly complex system with many parameters - which
> is pretty bad when the configuration impacts security, because regular
> users may not reaslise the consequences (and we'll get blamed for it).
> 
> Also, in my experience the deployments that really need this sort of
> encryption tend to be quite valuable, and the owners will be happy with
> higher hardware costs to compensate for the performance impact, if it
> gives them the feature. So even if the performance impact is 20% (worst
> case estimate), I'd say that may be acceptable.

Totally agree.

> I personally find the idea of encrypting tablespaces rather strange.
> Tablespaces are meant to define hysical location for objects, but this
> would also use them to "mark" objects as encrypted or not. That just
> seems misguided and would make the life harder for many users.
> 
> For example, what if I don't have any tablespaces (except for the
> default one), but I want to encrypt only some objects? Suddenly I have
> to create a tablespace, which will however cause various difficulties
> down the road (during pg_basebackup, etc.).

Yes, very good point.

> > In addition, while the 8k blocks would use a block cipher, the WAL would
> > likely use a stream cipher, and it will be very hard to use multiple
> > stream ciphers in a single WAL file.
> > 
> 
> Umm, why? Why would WAL necessarily use stream cipher instead of a block
> cipher with a suitable mode (say, CBC or XTS)? And even if it did use
> some stream cipter, why would it be hard to use multiple ciphers?

Well, the value of stream ciphers is that you can write as many bytes as
you want, rather than requiring all writes to be a multiple of the block
size.  The idea of having multiple tablespaces with different keys, and
switching streaming ciphers while encrypting only the part of the WAL
that needs it, and leaving the relfilenode unencrypted so we know which
keys to use, seems very complex.

> I mean, we'd probably want/need to start new streams for each WAL
> segment anyway, so that tools can read and process each WAL segment
> independently. So we wouldn't get very long streams anyway.

Well, 16MB quite a long stream, considering that AES256 is 128-bits or
16 bytes.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jun 13, 2019 at 07:49:48PM -0400, Bruce Momjian wrote:
>On Fri, Jun 14, 2019 at 12:41:17AM +0200, Tomas Vondra wrote:
>> On Thu, Jun 13, 2019 at 11:07:25AM -0400, Bruce Momjian wrote:
>> IMHO we should implement the simplest system possible, and optimize the
>> hell out of it without sacrificing any safety/security aspects. No smart
>> tunables, no extra GUCs to trade security for performance, nothing.
>>
>> Then once we have this working, we can see what the impact is, and make
>> informed choices based on that. It's really hard to make good choices
>> based on speculation, which is all we have at this point. And the danger
>> is we'll end up with overly complex system with many parameters - which
>> is pretty bad when the configuration impacts security, because regular
>> users may not reaslise the consequences (and we'll get blamed for it).
>>
>> Also, in my experience the deployments that really need this sort of
>> encryption tend to be quite valuable, and the owners will be happy with
>> higher hardware costs to compensate for the performance impact, if it
>> gives them the feature. So even if the performance impact is 20% (worst
>> case estimate), I'd say that may be acceptable.
>
>Totally agree.
>
>> I personally find the idea of encrypting tablespaces rather strange.
>> Tablespaces are meant to define hysical location for objects, but this
>> would also use them to "mark" objects as encrypted or not. That just
>> seems misguided and would make the life harder for many users.
>>
>> For example, what if I don't have any tablespaces (except for the
>> default one), but I want to encrypt only some objects? Suddenly I have
>> to create a tablespace, which will however cause various difficulties
>> down the road (during pg_basebackup, etc.).
>
>Yes, very good point.
>
>> > In addition, while the 8k blocks would use a block cipher, the WAL would
>> > likely use a stream cipher, and it will be very hard to use multiple
>> > stream ciphers in a single WAL file.
>> >
>>
>> Umm, why? Why would WAL necessarily use stream cipher instead of a block
>> cipher with a suitable mode (say, CBC or XTS)? And even if it did use
>> some stream cipter, why would it be hard to use multiple ciphers?
>
>Well, the value of stream ciphers is that you can write as many bytes as
>you want, rather than requiring all writes to be a multiple of the block
>size.  The idea of having multiple tablespaces with different keys, and
>switching streaming ciphers while encrypting only the part of the WAL
>that needs it, and leaving the relfilenode unencrypted so we know which
>keys to use, seems very complex.
>

OK, that makes sense.

FWIW my assumption was that we could just add an "encrypted" flag into
the main XLogRecord header, and then an extra part with important
encryption-related data - the key, and the important metadata needed by
external tools (e.g. relfilenode/block, needed by pg_waldump).

Then we wouldn't need to reshuffle the WAL, I think.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Fri, Jun 14, 2019 at 9:12 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Thu, Jun 13, 2019 at 07:49:48PM -0400, Bruce Momjian wrote:
> >On Fri, Jun 14, 2019 at 12:41:17AM +0200, Tomas Vondra wrote:
> >> On Thu, Jun 13, 2019 at 11:07:25AM -0400, Bruce Momjian wrote:
> >> IMHO we should implement the simplest system possible, and optimize the
> >> hell out of it without sacrificing any safety/security aspects. No smart
> >> tunables, no extra GUCs to trade security for performance, nothing.
> >>
> >> Then once we have this working, we can see what the impact is, and make
> >> informed choices based on that. It's really hard to make good choices
> >> based on speculation, which is all we have at this point. And the danger
> >> is we'll end up with overly complex system with many parameters - which
> >> is pretty bad when the configuration impacts security, because regular
> >> users may not reaslise the consequences (and we'll get blamed for it).
> >>
> >> Also, in my experience the deployments that really need this sort of
> >> encryption tend to be quite valuable, and the owners will be happy with
> >> higher hardware costs to compensate for the performance impact, if it
> >> gives them the feature. So even if the performance impact is 20% (worst
> >> case estimate), I'd say that may be acceptable.
> >
> >Totally agree.
> >
> >> I personally find the idea of encrypting tablespaces rather strange.
> >> Tablespaces are meant to define hysical location for objects, but this
> >> would also use them to "mark" objects as encrypted or not. That just
> >> seems misguided and would make the life harder for many users.
> >>
> >> For example, what if I don't have any tablespaces (except for the
> >> default one), but I want to encrypt only some objects? Suddenly I have
> >> to create a tablespace, which will however cause various difficulties
> >> down the road (during pg_basebackup, etc.).
> >
> >Yes, very good point.
> >
> >> > In addition, while the 8k blocks would use a block cipher, the WAL would
> >> > likely use a stream cipher, and it will be very hard to use multiple
> >> > stream ciphers in a single WAL file.
> >> >
> >>
> >> Umm, why? Why would WAL necessarily use stream cipher instead of a block
> >> cipher with a suitable mode (say, CBC or XTS)? And even if it did use
> >> some stream cipter, why would it be hard to use multiple ciphers?
> >
> >Well, the value of stream ciphers is that you can write as many bytes as
> >you want, rather than requiring all writes to be a multiple of the block
> >size.  The idea of having multiple tablespaces with different keys, and
> >switching streaming ciphers while encrypting only the part of the WAL
> >that needs it, and leaving the relfilenode unencrypted so we know which
> >keys to use, seems very complex.
> >
>
> OK, that makes sense.
>
> FWIW my assumption was that we could just add an "encrypted" flag into
> the main XLogRecord header, and then an extra part with important
> encryption-related data - the key, and the important metadata needed by
> external tools (e.g. relfilenode/block, needed by pg_waldump).
>
> Then we wouldn't need to reshuffle the WAL, I think.
>

Hmm, IIUC pg_waldump reads the main data containing the info for redo
operation of each rmgrs such as xl_heap_insert, xl_heap_update. I
wonder if we would need to doubly write the same data.



Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Jun 14, 2019 at 02:12:07AM +0200, Tomas Vondra wrote:
> FWIW my assumption was that we could just add an "encrypted" flag into
> the main XLogRecord header, and then an extra part with important
> encryption-related data - the key, and the important metadata needed by
> external tools (e.g. relfilenode/block, needed by pg_waldump).
> 
> Then we wouldn't need to reshuffle the WAL, I think.

I was thinking we would just encrypt the entire WAL file, and use the
WAL file name as the IV.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 6/13/19 11:07 AM, Bruce Momjian wrote:
> On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
>> Yeah, in principle since data key of 2 tier key architecture should
>> not go outside database I think we should not tell data keys to
>> utility commands. So the rearranging WAL format seems to be a better
>> solution but is there any reason why the main data is placed at end of
>> WAL record? I wonder if we can assemble WAL records as following order
>> and encrypt only 3 and 4.
>>
>> 1. Header data (XLogRecord and other headers)
>> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
>> 3. Block data (Tuple data, FPI)
>> 4. Sub data (e.g tuple data for logical decoding)
>
> Yes, that does sound like a reasonable idea.  It is similar to us not
> encrypting the clog --- there is little value.  However, if we only
> encrypt the cluster, we don't need to expose the relfilenode and we can
> just encrypt the entire WAL --- I like that simplicity.  We might find
> that the complexity of encrypting only certain tablespaces makes the
> system slower than just encrypting the entire cluster.


There are reasons other than performance to want more granular than
entire cluster encryption. Limiting the volume of encrypted data with
any one key for example. And not encrypting #1 & 2 above would help
avoid known-plaintext attacks I would think.


>> > > Also, for system catalog encryption, it could be a hard part. System
>> > > catalogs are initially created at initdb time and created by copying
>> > > from template1 when CREATE DATABASE. Therefore we would need to either
>> > > modify initdb so that it's aware of encryption keys and KMS or modify
>> > > database creation so that it copies database file while encrypting
>> > > them.
>> >
>> > I assume initdb will use the same API that you would use to start the
>> > server itself, e.g., type in a password, or contact a key server.
>>
>> I realized that in XTS encryption mode since we craft the tweak using
>> relfilenode we will need to have the different tweaks for system
>> catalogs in new database would change. So we might need to re-encrypt
>> system catalogs when CREATE DATABASE after all. I suspect that even
>> the cluster-wide encryption has the same problem.
>
> Yes, this is why I want to just do cluster-wide encryption at this
> stage.
>
> In addition, while the 8k blocks would use a block cipher, the WAL would
> likely use a stream cipher, and it will be very hard to use multiple
> stream ciphers in a single WAL file.


I don't understand why we would not just use AES for both.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote:
> On 6/13/19 11:07 AM, Bruce Momjian wrote:
> > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
> >> Yeah, in principle since data key of 2 tier key architecture should
> >> not go outside database I think we should not tell data keys to
> >> utility commands. So the rearranging WAL format seems to be a better
> >> solution but is there any reason why the main data is placed at end of
> >> WAL record? I wonder if we can assemble WAL records as following order
> >> and encrypt only 3 and 4.
> >> 
> >> 1. Header data (XLogRecord and other headers)
> >> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
> >> 3. Block data (Tuple data, FPI)
> >> 4. Sub data (e.g tuple data for logical decoding)
> > 
> > Yes, that does sound like a reasonable idea.  It is similar to us not
> > encrypting the clog --- there is little value.  However, if we only
> > encrypt the cluster, we don't need to expose the relfilenode and we can
> > just encrypt the entire WAL --- I like that simplicity.  We might find
> > that the complexity of encrypting only certain tablespaces makes the
> > system slower than just encrypting the entire cluster.
> 
> 
> There are reasons other than performance to want more granular than
> entire cluster encryption. Limiting the volume of encrypted data with
> any one key for example. And not encrypting #1 & 2 above would help
> avoid known-plaintext attacks I would think.
> 
> 
> >> > > Also, for system catalog encryption, it could be a hard part. System
> >> > > catalogs are initially created at initdb time and created by copying
> >> > > from template1 when CREATE DATABASE. Therefore we would need to either
> >> > > modify initdb so that it's aware of encryption keys and KMS or modify
> >> > > database creation so that it copies database file while encrypting
> >> > > them.
> >> >
> >> > I assume initdb will use the same API that you would use to start the
> >> > server itself, e.g., type in a password, or contact a key server.
> >> 
> >> I realized that in XTS encryption mode since we craft the tweak using
> >> relfilenode we will need to have the different tweaks for system
> >> catalogs in new database would change. So we might need to re-encrypt
> >> system catalogs when CREATE DATABASE after all. I suspect that even
> >> the cluster-wide encryption has the same problem.
> > 
> > Yes, this is why I want to just do cluster-wide encryption at this
> > stage.
> > 
> > In addition, while the 8k blocks would use a block cipher, the WAL would
> > likely use a stream cipher, and it will be very hard to use multiple
> > stream ciphers in a single WAL file.
> 
> 
> I don't understand why we would not just use AES for both.

Uh, AES is an encryption cipher.  You can use it with block mode, CBC,
or stream mode, CTR, GCM;  see:

    http://momjian.us/main/writings/crypto.pdf#page=7

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 6/14/19 6:09 PM, Bruce Momjian wrote:
> On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote:
>> On 6/13/19 11:07 AM, Bruce Momjian wrote:
>> > In addition, while the 8k blocks would use a block cipher, the WAL would
>> > likely use a stream cipher, and it will be very hard to use multiple
>> > stream ciphers in a single WAL file.
>>
>> I don't understand why we would not just use AES for both.
>
> Uh, AES is an encryption cipher.  You can use it with block mode, CBC,
> or stream mode, CTR, GCM;  see:


AES is a block cipher, not a stream cipher. Yes you can use it in
different modes, including chained modes (and CBC is what I would pick),
but I assumed you were talking about an actual stream cipher algorithm.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Fri, Jun 14, 2019 at 09:37:37PM -0400, Joe Conway wrote:
> On 6/14/19 6:09 PM, Bruce Momjian wrote:
> > On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote:
> >> On 6/13/19 11:07 AM, Bruce Momjian wrote:
> >> > In addition, while the 8k blocks would use a block cipher, the WAL would
> >> > likely use a stream cipher, and it will be very hard to use multiple
> >> > stream ciphers in a single WAL file.
> >> 
> >> I don't understand why we would not just use AES for both.
> > 
> > Uh, AES is an encryption cipher.  You can use it with block mode, CBC,
> > or stream mode, CTR, GCM;  see:
> 
> 
> AES is a block cipher, not a stream cipher. Yes you can use it in
> different modes, including chained modes (and CBC is what I would pick),
> but I assumed you were talking about an actual stream cipher algorithm.

OK, to be specific, I was thinking of using aes128-cbc for the 8k pages
and aes128-ctr for the WAL.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote:
> On 6/13/19 11:07 AM, Bruce Momjian wrote:
> > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
> >> Yeah, in principle since data key of 2 tier key architecture should
> >> not go outside database I think we should not tell data keys to
> >> utility commands. So the rearranging WAL format seems to be a better
> >> solution but is there any reason why the main data is placed at end of
> >> WAL record? I wonder if we can assemble WAL records as following order
> >> and encrypt only 3 and 4.
> >> 
> >> 1. Header data (XLogRecord and other headers)
> >> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
> >> 3. Block data (Tuple data, FPI)
> >> 4. Sub data (e.g tuple data for logical decoding)
> > 
> > Yes, that does sound like a reasonable idea.  It is similar to us not
> > encrypting the clog --- there is little value.  However, if we only
> > encrypt the cluster, we don't need to expose the relfilenode and we can
> > just encrypt the entire WAL --- I like that simplicity.  We might find
> > that the complexity of encrypting only certain tablespaces makes the
> > system slower than just encrypting the entire cluster.
> 
> 
> There are reasons other than performance to want more granular than
> entire cluster encryption. Limiting the volume of encrypted data with
> any one key for example. And not encrypting #1 & 2 above would help
> avoid known-plaintext attacks I would think.

There are no known non-exhaustive plaintext attacks on AES:

    https://crypto.stackexchange.com/questions/1512/why-is-aes-resistant-to-known-plaintext-attacks

Even if we don't encrypt the first part of the WAL record (1 & 2), the
block data (3) probably has enough format for a plaintext attack.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 6/15/19 9:28 PM, Bruce Momjian wrote:
> On Fri, Jun 14, 2019 at 02:27:17PM -0400, Joe Conway wrote:
>> On 6/13/19 11:07 AM, Bruce Momjian wrote:
>> > On Thu, Jun 13, 2019 at 04:26:47PM +0900, Masahiko Sawada wrote:
>> >> Yeah, in principle since data key of 2 tier key architecture should
>> >> not go outside database I think we should not tell data keys to
>> >> utility commands. So the rearranging WAL format seems to be a better
>> >> solution but is there any reason why the main data is placed at end of
>> >> WAL record? I wonder if we can assemble WAL records as following order
>> >> and encrypt only 3 and 4.
>> >>
>> >> 1. Header data (XLogRecord and other headers)
>> >> 2. Main data (xl_heap_insert, xl_heap_update etc + related data)
>> >> 3. Block data (Tuple data, FPI)
>> >> 4. Sub data (e.g tuple data for logical decoding)
>> >
>> > Yes, that does sound like a reasonable idea.  It is similar to us not
>> > encrypting the clog --- there is little value.  However, if we only
>> > encrypt the cluster, we don't need to expose the relfilenode and we can
>> > just encrypt the entire WAL --- I like that simplicity.  We might find
>> > that the complexity of encrypting only certain tablespaces makes the
>> > system slower than just encrypting the entire cluster.
>>
>>
>> There are reasons other than performance to want more granular than
>> entire cluster encryption. Limiting the volume of encrypted data with
>> any one key for example. And not encrypting #1 & 2 above would help
>> avoid known-plaintext attacks I would think.
>
> There are no known non-exhaustive plaintext attacks on AES:
>
>     https://crypto.stackexchange.com/questions/1512/why-is-aes-resistant-to-known-plaintext-attacks

Even that non-authoritative stackexchange thread has varying opinions.
Surely you don't claim that limiting know plaintext as much as is
practical is a bad idea in general.

> Even if we don't encrypt the first part of the WAL record (1 & 2), the
> block data (3) probably has enough format for a plaintext attack.

Perhaps.

In any case it doesn't address my first point, which is limiting the
volume encrypted with the same key. Another valid reason is you might
have data at varying sensitivity levels and prefer different keys be
used for each level.

And although I'm not proposing this for the first implementation, yet
another reason is I might want to additionally control "transparent
access" to data based on who is logged in. That could be done by
layering an additional key on top of the per-tablespace key for example.

The bottom line in my mind is encrypting the entire database with a
single key is not much different/better than using filesystem
encryption, so I'm not sure it is worth the effort and complexity to get
that capability. I think having the ability to encrypt at the tablespace
level adds a lot of capability for minimal extra complexity.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> On 6/15/19 9:28 PM, Bruce Momjian wrote:
> >> There are reasons other than performance to want more granular than
> >> entire cluster encryption. Limiting the volume of encrypted data with
> >> any one key for example. And not encrypting #1 & 2 above would help
> >> avoid known-plaintext attacks I would think.
> > 
> > There are no known non-exhaustive plaintext attacks on AES:
> > 
> >     https://crypto.stackexchange.com/questions/1512/why-is-aes-resistant-to-known-plaintext-attacks
> 
> Even that non-authoritative stackexchange thread has varying opinions.
> Surely you don't claim that limiting know plaintext as much as is
> practical is a bad idea in general.

I think we have to look at complexity vs. benefit.

> > Even if we don't encrypt the first part of the WAL record (1 & 2), the
> > block data (3) probably has enough format for a plaintext attack.
> 
> Perhaps.
> 
> In any case it doesn't address my first point, which is limiting the
> volume encrypted with the same key. Another valid reason is you might
> have data at varying sensitivity levels and prefer different keys be
> used for each level.

That seems quite complex.

> And although I'm not proposing this for the first implementation, yet
> another reason is I might want to additionally control "transparent
> access" to data based on who is logged in. That could be done by
> layering an additional key on top of the per-tablespace key for example.
> 
> The bottom line in my mind is encrypting the entire database with a
> single key is not much different/better than using filesystem
> encryption, so I'm not sure it is worth the effort and complexity to get
> that capability. I think having the ability to encrypt at the tablespace
> level adds a lot of capability for minimal extra complexity.

I disagree.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sun, Jun 16, 2019 at 09:45:09AM -0400, Bruce Momjian wrote:
> On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> > And although I'm not proposing this for the first implementation, yet
> > another reason is I might want to additionally control "transparent
> > access" to data based on who is logged in. That could be done by
> > layering an additional key on top of the per-tablespace key for example.
> > 
> > The bottom line in my mind is encrypting the entire database with a
> > single key is not much different/better than using filesystem
> > encryption, so I'm not sure it is worth the effort and complexity to get
> > that capability. I think having the ability to encrypt at the tablespace
> > level adds a lot of capability for minimal extra complexity.
> 
> I disagree.

I will add that OpenSSL has been removing features and compatibility
because the added complexity had hidden exploits that they could not
have anticipated.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 6/16/19 9:45 AM, Bruce Momjian wrote:
> On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
>> In any case it doesn't address my first point, which is limiting the
>> volume encrypted with the same key. Another valid reason is you might
>> have data at varying sensitivity levels and prefer different keys be
>> used for each level.
>
> That seems quite complex.


How? It is no more complex than encrypting at the tablespace level
already gives you - in that case you get this property for free if you
care to use it.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 6/16/19 9:46 AM, Bruce Momjian wrote:
> On Sun, Jun 16, 2019 at 09:45:09AM -0400, Bruce Momjian wrote:
>> On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
>> > And although I'm not proposing this for the first implementation, yet
>> > another reason is I might want to additionally control "transparent
>> > access" to data based on who is logged in. That could be done by
>> > layering an additional key on top of the per-tablespace key for example.
>> >
>> > The bottom line in my mind is encrypting the entire database with a
>> > single key is not much different/better than using filesystem
>> > encryption, so I'm not sure it is worth the effort and complexity to get
>> > that capability. I think having the ability to encrypt at the tablespace
>> > level adds a lot of capability for minimal extra complexity.
>>
>> I disagree.
>
> I will add that OpenSSL has been removing features and compatibility
> because the added complexity had hidden exploits that they could not
> have anticipated.

Sorry but I'm not buying it.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Greetings,

* Joe Conway (mail@joeconway.com) wrote:
> On 6/16/19 9:45 AM, Bruce Momjian wrote:
> > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> >> In any case it doesn't address my first point, which is limiting the
> >> volume encrypted with the same key. Another valid reason is you might
> >> have data at varying sensitivity levels and prefer different keys be
> >> used for each level.
> >
> > That seems quite complex.
>
> How? It is no more complex than encrypting at the tablespace level
> already gives you - in that case you get this property for free if you
> care to use it.

Perhaps not surprising, but I'm definitely in agreement with Joe
regarding having multiple keys when possible and (reasonably)
straight-forward to do so.  I also don't buy off on the OpenSSL
argument; their more severe issues certainly haven't been due to key
management issues such as what we're discussing here, so I don't think
the argument applies.

Thanks,

Stephen

On Sun, Jun 16, 2019 at 12:42:55PM -0400, Joe Conway wrote:
> On 6/16/19 9:45 AM, Bruce Momjian wrote:
> > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> >> In any case it doesn't address my first point, which is limiting the
> >> volume encrypted with the same key. Another valid reason is you might
> >> have data at varying sensitivity levels and prefer different keys be
> >> used for each level.
> > 
> > That seems quite complex.
> 
> 
> How? It is no more complex than encrypting at the tablespace level
> already gives you - in that case you get this property for free if you
> care to use it.

All keys used to encrypt WAL data must be unlocked at all times or crash
recovery, PITR, and replication will not stop when it hits a locked key.
Given that, how much value is there in allowing a key per tablespace?

I don't see how this is better than telling users they have to create a
separate cluster per key.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Sun, Jun 16, 2019 at 12:42:55PM -0400, Joe Conway wrote:
> > On 6/16/19 9:45 AM, Bruce Momjian wrote:
> > > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> > >> In any case it doesn't address my first point, which is limiting the
> > >> volume encrypted with the same key. Another valid reason is you might
> > >> have data at varying sensitivity levels and prefer different keys be
> > >> used for each level.
> > >
> > > That seems quite complex.
> >
> > How? It is no more complex than encrypting at the tablespace level
> > already gives you - in that case you get this property for free if you
> > care to use it.
>
> All keys used to encrypt WAL data must be unlocked at all times or crash
> recovery, PITR, and replication will not stop when it hits a locked key.
> Given that, how much value is there in allowing a key per tablespace?

There's a few different things to discuss here, admittedly, but I don't
think it means that there's no value in having a key per tablespace.

Ideally, a given backend would only need, and only have access to, the
keys for the tablespaces that it is allowed to operate on.  I realize
that's a bit farther than what we're talking about today, but hopefully
not too much to be able to consider.

Thanks,

Stephen

On Sun, Jun 16, 2019 at 02:10:23PM -0400, Stephen Frost wrote:
>Greetings,
>
>* Joe Conway (mail@joeconway.com) wrote:
>> On 6/16/19 9:45 AM, Bruce Momjian wrote:
>> > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
>> >> In any case it doesn't address my first point, which is limiting the
>> >> volume encrypted with the same key. Another valid reason is you might
>> >> have data at varying sensitivity levels and prefer different keys be
>> >> used for each level.
>> >
>> > That seems quite complex.
>>
>> How? It is no more complex than encrypting at the tablespace level
>> already gives you - in that case you get this property for free if you
>> care to use it.
>
>Perhaps not surprising, but I'm definitely in agreement with Joe
>regarding having multiple keys when possible and (reasonably)
>straight-forward to do so.  I also don't buy off on the OpenSSL
>argument; their more severe issues certainly haven't been due to key
>management issues such as what we're discussing here, so I don't think
>the argument applies.
>

I'm not sure what exactly is the "OpenSSL argument" you're disagreeing
with? IMHO Bruce is quite right that the risk of vulnerabilities grows
with the complexity of the system (both due to implementation bugs and
general design weaknesses). I don't think it's tied to the key
management specifically, except that it's one of the parts that may
contribute to the complexity.

(It's often claimed that key management is one of the weakest points of
current crypto systems - we have safe (a)symmetric algorithms, but safe
handling of keys is an issue. I don't have data / papers supporting this
claim, I kinda believe it.)

Now, I'm not opposed to eventually implementing something more
elaborate, but I also think just encrypting the whole cluster (not
necessarily with a single key, but with one master key) would be enough
for vast majority of users. Plus it's less error prone and easier to
operate (backups, replicas, crash recovery, ...).

But there's about 0% chance we'll get that in v1, of course, so we need
s "minimum viable product" to build on anyway.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> On Sun, Jun 16, 2019 at 02:10:23PM -0400, Stephen Frost wrote:
> >* Joe Conway (mail@joeconway.com) wrote:
> >>On 6/16/19 9:45 AM, Bruce Momjian wrote:
> >>> On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> >>>> In any case it doesn't address my first point, which is limiting the
> >>>> volume encrypted with the same key. Another valid reason is you might
> >>>> have data at varying sensitivity levels and prefer different keys be
> >>>> used for each level.
> >>>
> >>> That seems quite complex.
> >>
> >>How? It is no more complex than encrypting at the tablespace level
> >>already gives you - in that case you get this property for free if you
> >>care to use it.
> >
> >Perhaps not surprising, but I'm definitely in agreement with Joe
> >regarding having multiple keys when possible and (reasonably)
> >straight-forward to do so.  I also don't buy off on the OpenSSL
> >argument; their more severe issues certainly haven't been due to key
> >management issues such as what we're discussing here, so I don't think
> >the argument applies.
>
> I'm not sure what exactly is the "OpenSSL argument" you're disagreeing
> with? IMHO Bruce is quite right that the risk of vulnerabilities grows
> with the complexity of the system (both due to implementation bugs and
> general design weaknesses). I don't think it's tied to the key
> management specifically, except that it's one of the parts that may
> contribute to the complexity.

While I understand that complexity of the system can lead to
vulnerabilities, I don't agree that it's appropriate or sensible to
equate the ones that OpenSSL has had to deal with as being similar to
what we might have to deal with, given this additional proposed
complexity.

> (It's often claimed that key management is one of the weakest points of
> current crypto systems - we have safe (a)symmetric algorithms, but safe
> handling of keys is an issue. I don't have data / papers supporting this
> claim, I kinda believe it.)

Yes, I agree entirely that key management is absolutely one of the
hardest things to get right in crypto systems.  It is, however, not what
the OpenSSL issues have been about and so while I agree that we may have
some bugs there, it's not fair to say that they're equivilant to what
OpenSSL has been dealing with.

> Now, I'm not opposed to eventually implementing something more
> elaborate, but I also think just encrypting the whole cluster (not
> necessarily with a single key, but with one master key) would be enough
> for vast majority of users. Plus it's less error prone and easier to
> operate (backups, replicas, crash recovery, ...).

I agree that it'd be better than nothing but if we have the opportunity
now to introduce what would hopefully be a relatively small capability,
then I'm certainly all for it.

> But there's about 0% chance we'll get that in v1, of course, so we need
> s "minimum viable product" to build on anyway.

There seems like a whole lot of space between something very elaborate
and only supporting one key.

Thanks,

Stephen

On 6/17/19 8:12 AM, Stephen Frost wrote:
>> But there's about 0% chance we'll get that in v1, of course, so we need
>> s "minimum viable product" to build on anyway.
> 
> There seems like a whole lot of space between something very elaborate
> and only supporting one key.

I think this is exactly the point -- IMHO one key per tablespace is a
nice and very sensible compromise. I can imagine all kinds of more
complex things that would be "nice to have" but that gets us most of the
flexibility needed with minimal additional complexity.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Fri, Jun 14, 2019 at 7:41 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> I personally find the idea of encrypting tablespaces rather strange.
> Tablespaces are meant to define hysical location for objects, but this
> would also use them to "mark" objects as encrypted or not. That just
> seems misguided and would make the life harder for many users.
>
> For example, what if I don't have any tablespaces (except for the
> default one), but I want to encrypt only some objects? Suddenly I have
> to create a tablespace, which will however cause various difficulties
> down the road (during pg_basebackup, etc.).

I guess that we can have an encrypted tabelspace by default (e.g.
pg_default_enc). Or we encrypt per tables while having encryption keys
per tablespaces.

On Mon, Jun 17, 2019 at 6:54 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Sun, Jun 16, 2019 at 02:10:23PM -0400, Stephen Frost wrote:
> >Greetings,
> >
> >* Joe Conway (mail@joeconway.com) wrote:
> >> On 6/16/19 9:45 AM, Bruce Momjian wrote:
> >> > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> >> >> In any case it doesn't address my first point, which is limiting the
> >> >> volume encrypted with the same key. Another valid reason is you might
> >> >> have data at varying sensitivity levels and prefer different keys be
> >> >> used for each level.
> >> >
> >> > That seems quite complex.
> >>
> >> How? It is no more complex than encrypting at the tablespace level
> >> already gives you - in that case you get this property for free if you
> >> care to use it.
> >
> >Perhaps not surprising, but I'm definitely in agreement with Joe
> >regarding having multiple keys when possible and (reasonably)
> >straight-forward to do so.  I also don't buy off on the OpenSSL
> >argument; their more severe issues certainly haven't been due to key
> >management issues such as what we're discussing here, so I don't think
> >the argument applies.
> >
>
> I'm not sure what exactly is the "OpenSSL argument" you're disagreeing
> with? IMHO Bruce is quite right that the risk of vulnerabilities grows
> with the complexity of the system (both due to implementation bugs and
> general design weaknesses). I don't think it's tied to the key
> management specifically, except that it's one of the parts that may
> contribute to the complexity.
>
> (It's often claimed that key management is one of the weakest points of
> current crypto systems - we have safe (a)symmetric algorithms, but safe
> handling of keys is an issue. I don't have data / papers supporting this
> claim, I kinda believe it.)
>
> Now, I'm not opposed to eventually implementing something more
> elaborate, but I also think just encrypting the whole cluster (not
> necessarily with a single key, but with one master key) would be enough
> for vast majority of users. Plus it's less error prone and easier to
> operate (backups, replicas, crash recovery, ...).
>
> But there's about 0% chance we'll get that in v1, of course, so we need
> s "minimum viable product" to build on anyway.
>

I agree that we need minimum viable product first. But I'm not sure
it's true that the implementing the cluster-wide TDE first could be
the first step of per-tablespace/table TDE.

The purpose of cluster-wide TDE and table/tablespace TDE are slightly
different in terms of encryption target objects. The cluster-wide TDE
would be a good solution for users who want to encrypt everything
while the table/tabelspace TDE would help more severe use cases in
terms of both of security and performance.

The cluster-wide TDE eventually encrypts SLRU data and all WAL
including non-user data related WAL while table/tablespace TDE doesn't
unless we develop such functionality. In addition, the cluster-wide
TDE also encrypts system catalogs but in table/tablespace TDE user
would be able to control that somewhat. That is, if we developed the
cluster-wide TDE first, when we develop table/tablespace TDE on top of
that we would need to change TDE so that table/tablespace TDE can
encrypt even non-user data related data while retaining its simple
user interface, which would rather make the feature complex, I'm
concerned. We can support them as different TDE features but I'm not
sure it's a good choice for users.

From perspective of  cryptographic, I think the fine grained TDE would
be better solution. Therefore if we eventually want the fine grained
TDE I wonder if it might be better to develop the table/tablespace TDE
first while keeping it simple as much as possible in v1, and then we
can provide the functionality to encrypt other data in database
cluster to satisfy the encrypting-everything requirement. I guess that
it's easier to incrementally add encryption target objects rather than
making it fine grained while not changing encryption target objects.

FWIW I'm writing a draft patch of per tablespace TDE and will submit
it in this month. We can more discuss the complexity of the proposed
TDE using it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 6/17/19 8:29 AM, Masahiko Sawada wrote:
> From perspective of  cryptographic, I think the fine grained TDE would
> be better solution. Therefore if we eventually want the fine grained
> TDE I wonder if it might be better to develop the table/tablespace TDE
> first while keeping it simple as much as possible in v1, and then we
> can provide the functionality to encrypt other data in database
> cluster to satisfy the encrypting-everything requirement. I guess that
> it's easier to incrementally add encryption target objects rather than
> making it fine grained while not changing encryption target objects.
> 
> FWIW I'm writing a draft patch of per tablespace TDE and will submit
> it in this month. We can more discuss the complexity of the proposed
> TDE using it.

+1

Looking forward to it.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The cluster-wide TDE eventually encrypts SLRU data and all WAL
> including non-user data related WAL while table/tablespace TDE doesn't
> unless we develop such functionality. In addition, the cluster-wide
> TDE also encrypts system catalogs but in table/tablespace TDE user
> would be able to control that somewhat. That is, if we developed the
> cluster-wide TDE first, when we develop table/tablespace TDE on top of
> that we would need to change TDE so that table/tablespace TDE can
> encrypt even non-user data related data while retaining its simple
> user interface, which would rather make the feature complex, I'm
> concerned.

Isn't this only a problem of pg_upgrade? If the whole instance (including
catalog) is encrypted and user wants to adopt the table/tablespace TDE, then
pg_upgrade can simply decrypt the catalog, plus tables/tablespaces which
should no longer be encrypted. Conversely, if only some tables/tablespaces are
encrypted and user wants to encrypt the whole cluster, pg_upgrade will encrypt
the non-encrypted files.

> We can support them as different TDE features but I'm not sure it's a good
> choice for users.

IMO it does not matter which approach (cluster vs table/tablespace) is
implemented first. What matters is to design the user interface so that
addition of the other of the two features does not make users confused.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Antonin Houska <ah@cybertec.at> wrote:

> Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> 
> > The cluster-wide TDE eventually encrypts SLRU data and all WAL
> > including non-user data related WAL while table/tablespace TDE doesn't
> > unless we develop such functionality. In addition, the cluster-wide
> > TDE also encrypts system catalogs but in table/tablespace TDE user
> > would be able to control that somewhat. That is, if we developed the
> > cluster-wide TDE first, when we develop table/tablespace TDE on top of
> > that we would need to change TDE so that table/tablespace TDE can
> > encrypt even non-user data related data while retaining its simple
> > user interface, which would rather make the feature complex, I'm
> > concerned.
> 
> Isn't this only a problem of pg_upgrade?

Sorry, this is not a use case for pg_upgrade. Rather it's about a separate
encryption/decryption utility.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote:
>On 6/17/19 8:29 AM, Masahiko Sawada wrote:
>> From perspective of  cryptographic, I think the fine grained TDE would
>> be better solution. Therefore if we eventually want the fine grained
>> TDE I wonder if it might be better to develop the table/tablespace TDE
>> first while keeping it simple as much as possible in v1, and then we
>> can provide the functionality to encrypt other data in database
>> cluster to satisfy the encrypting-everything requirement. I guess that
>> it's easier to incrementally add encryption target objects rather than
>> making it fine grained while not changing encryption target objects.
>>
>> FWIW I'm writing a draft patch of per tablespace TDE and will submit
>> it in this month. We can more discuss the complexity of the proposed
>> TDE using it.
>
>+1
>
>Looking forward to it.
>

Yep. In particular, I'm interested in those aspects:

(1) What's the proposed minimum viable product, and how do we expect to
extend it with the more elaborate features. I don't expect perfect
specification, but we should have some idea so that we don't paint
ourselves in the corner.

(2) How does it affect recovery, backups and replication (both physical
and logical)? That is, which other parts need to know the encryption keys
to function properly?

(3) What does it mean for external tools (pg_waldump, pg_upgrade,
pg_rewind etc.)? 


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Jun 17, 2019 at 08:29:02AM -0400, Joe Conway wrote:
>On 6/17/19 8:12 AM, Stephen Frost wrote:
>>> But there's about 0% chance we'll get that in v1, of course, so we need
>>> s "minimum viable product" to build on anyway.
>>
>> There seems like a whole lot of space between something very elaborate
>> and only supporting one key.
>
>I think this is exactly the point -- IMHO one key per tablespace is a
>nice and very sensible compromise. I can imagine all kinds of more
>complex things that would be "nice to have" but that gets us most of the
>flexibility needed with minimal additional complexity.
>

Not sure.

I think it's clear the main challenge is encryption of shared resources,
particularly WAL (when each WAL record gets encrypted with the same key as
the object). Considering the importance of WAL, that complicates all sorts
of stuff (recovery, replication, ...).

Encrypting WAL with a single key would be way easier, because we could
(probably) just encrypt whole WAL pages. That may not be appropriate for
some advanced use cases, of course. It would work when used as a db-level
replacement for FDE, which I think was the primary motivation for TDE.

In any case, if we end up with a more complex/advanced design, I've
already voiced my opinion that binding the keys to tablespaces is the
wrong abstraction, and I think we'll regret it eventually. For example,
why have we invented publications instead of using tablespaces?


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> In any case, if we end up with a more complex/advanced design, I've
> already voiced my opinion that binding the keys to tablespaces is the
> wrong abstraction, and I think we'll regret it eventually. For example,
> why have we invented publications instead of using tablespaces?

I would certainly hope that we don't stop at tablespaces, they just seem
like a much simpler piece to bite off piece than going to table-level
right off, and they make sense for some environments where there's a
relatively small number of levels of separation, which are already being
segregated into different filesystems (or at least directories) for the
same reason that you want different encryption keys.

Thanks,

Stephen

On Mon, Jun 17, 2019 at 10:33:11AM -0400, Stephen Frost wrote:
>Greetings,
>
>* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
>> In any case, if we end up with a more complex/advanced design, I've
>> already voiced my opinion that binding the keys to tablespaces is the
>> wrong abstraction, and I think we'll regret it eventually. For example,
>> why have we invented publications instead of using tablespaces?
>
>I would certainly hope that we don't stop at tablespaces, they just seem
>like a much simpler piece to bite off piece than going to table-level
>right off, and they make sense for some environments where there's a
>relatively small number of levels of separation, which are already being
>segregated into different filesystems (or at least directories) for the
>same reason that you want different encryption keys.
>

Why not to use the right abstraction from the beginning? I already
mentioned publications, which I think we can use as an inspiration. So
it's not like this would be a major design task, IMHO.

In my experience it's pretty difficult to change abstractions the design
is based on, not just because it tends to be invasive implementation-wise,
but also because users get used to it.

FWIW this is one of the reasons why I advocate for v1 not to allow this,
because it's much easier to extend the design

    single group -> multiple groups

compared to

    one way to group objects -> different way to group objects



regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Jun 18, 2019 at 5:07 PM shawn wang <shawn.wang.pg@gmail.com> wrote:
>
> Masahiko Sawada <sawada.mshk@gmail.com> 于2019年6月17日周一 下午8:30写道：
>>
>> On Fri, Jun 14, 2019 at 7:41 AM Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> > I personally find the idea of encrypting tablespaces rather strange.
>> > Tablespaces are meant to define hysical location for objects, but this
>> > would also use them to "mark" objects as encrypted or not. That just
>> > seems misguided and would make the life harder for many users.
>> >
>> > For example, what if I don't have any tablespaces (except for the
>> > default one), but I want to encrypt only some objects? Suddenly I have
>> > to create a tablespace, which will however cause various difficulties
>> > down the road (during pg_basebackup, etc.).
>>
>> I guess that we can have an encrypted tabelspace by default (e.g.
>> pg_default_enc). Or we encrypt per tables while having encryption keys
>> per tablespaces.
>
>
> Hi Sawada-san,
> I do agree with it.
>>
>>
>> On Mon, Jun 17, 2019 at 6:54 AM Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> >
>> > On Sun, Jun 16, 2019 at 02:10:23PM -0400, Stephen Frost wrote:
>> > >Greetings,
>> > >
>> > >* Joe Conway (mail@joeconway.com) wrote:
>> > >> On 6/16/19 9:45 AM, Bruce Momjian wrote:
>> > >> > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
>> > >> >> In any case it doesn't address my first point, which is limiting the
>> > >> >> volume encrypted with the same key. Another valid reason is you might
>> > >> >> have data at varying sensitivity levels and prefer different keys be
>> > >> >> used for each level.
>> > >> >
>> > >> > That seems quite complex.
>> > >>
>> > >> How? It is no more complex than encrypting at the tablespace level
>> > >> already gives you - in that case you get this property for free if you
>> > >> care to use it.
>> > >
>> > >Perhaps not surprising, but I'm definitely in agreement with Joe
>> > >regarding having multiple keys when possible and (reasonably)
>> > >straight-forward to do so.  I also don't buy off on the OpenSSL
>> > >argument; their more severe issues certainly haven't been due to key
>> > >management issues such as what we're discussing here, so I don't think
>> > >the argument applies.
>> > >
>> >
>> > I'm not sure what exactly is the "OpenSSL argument" you're disagreeing
>> > with? IMHO Bruce is quite right that the risk of vulnerabilities grows
>> > with the complexity of the system (both due to implementation bugs and
>> > general design weaknesses). I don't think it's tied to the key
>> > management specifically, except that it's one of the parts that may
>> > contribute to the complexity.
>> >
>> > (It's often claimed that key management is one of the weakest points of
>> > current crypto systems - we have safe (a)symmetric algorithms, but safe
>> > handling of keys is an issue. I don't have data / papers supporting this
>> > claim, I kinda believe it.)
>> >
>> > Now, I'm not opposed to eventually implementing something more
>> > elaborate, but I also think just encrypting the whole cluster (not
>> > necessarily with a single key, but with one master key) would be enough
>> > for vast majority of users. Plus it's less error prone and easier to
>> > operate (backups, replicas, crash recovery, ...).
>> >
>> > But there's about 0% chance we'll get that in v1, of course, so we need
>> > s "minimum viable product" to build on anyway.
>> >
>>
>> I agree that we need minimum viable product first. But I'm not sure
>> it's true that the implementing the cluster-wide TDE first could be
>> the first step of per-tablespace/table TDE.
>
>
> Yes, we could complete the per-tablespace/table TDE in version 13.
> And we could do cluster-wide TDE in the next version.
> But I remember you said there are so many keys to manage in the table-level.

I think even if we provide the per table encryption we can have
encryption keys per tablepaces. That is, all tables on the same
tablespace encryption use the same encryption keys but user can
control encrypted objects per tables.

> Will we add the table-level TDE in the first version?

I hope so but It's under discussion now.

> And I have two questions.
> 1. Will we add hooks to support replacing the encryption algorithms?
> 2. Will we add some encryption algorithm or use some in some libraries?

Currently the WIP patch uses openssl and supports only AES-256 and I
don't have a plan to have such extensibility for now. But it might be
a good idea in the future. I think it would be not hard to support
symmetric encryption altgorithms supported by openssl but would you
like to support other encryption algorithms?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 6/20/19 8:34 AM, Masahiko Sawada wrote:
> I think even if we provide the per table encryption we can have
> encryption keys per tablepaces. That is, all tables on the same
> tablespace encryption use the same encryption keys but user can
> control encrypted objects per tables.
> 
>> Will we add the table-level TDE in the first version?
> 
> I hope so but It's under discussion now.

+1

>> And I have two questions.
>> 1. Will we add hooks to support replacing the encryption algorithms?
>> 2. Will we add some encryption algorithm or use some in some libraries?
> 
> Currently the WIP patch uses openssl and supports only AES-256 and I
> don't have a plan to have such extensibility for now. But it might be
> a good idea in the future. I think it would be not hard to support
> symmetric encryption altgorithms supported by openssl but would you
> like to support other encryption algorithms?

Supporting other symmetric encryption algorithms would be nice but I
don't think that is required for the first version. It would also be
nice but not initially required to support different encryption
libraries. The implementation should be written with both of these
eventualities in mind though IMHO.

I would like to see strategically placed hooks in the key management so
that an extension could, for example, layer another key in between the
master key and the per-tablespace key. This would allow extensions to
provide additional control over when decryption is allowed.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Thu, Jun 20, 2019 at 10:46 PM Joe Conway <mail@joeconway.com> wrote:
>
> On 6/20/19 8:34 AM, Masahiko Sawada wrote:
> > I think even if we provide the per table encryption we can have
> > encryption keys per tablepaces. That is, all tables on the same
> > tablespace encryption use the same encryption keys but user can
> > control encrypted objects per tables.
> >
> >> Will we add the table-level TDE in the first version?
> >
> > I hope so but It's under discussion now.
>
> +1
>
> >> And I have two questions.
> >> 1. Will we add hooks to support replacing the encryption algorithms?
> >> 2. Will we add some encryption algorithm or use some in some libraries?
> >
> > Currently the WIP patch uses openssl and supports only AES-256 and I
> > don't have a plan to have such extensibility for now. But it might be
> > a good idea in the future. I think it would be not hard to support
> > symmetric encryption altgorithms supported by openssl but would you
> > like to support other encryption algorithms?
>
> Supporting other symmetric encryption algorithms would be nice but I
> don't think that is required for the first version. It would also be
> nice but not initially required to support different encryption
> libraries. The implementation should be written with both of these
> eventualities in mind though IMHO.

Agreed.

>
> I would like to see strategically placed hooks in the key management so
> that an extension could, for example, layer another key in between the
> master key and the per-tablespace key. This would allow extensions to
> provide additional control over when decryption is allowed.

Interesting.

The master key management is a important topic. In my proposal, we
provide generic key management APIs such as getkey, removekey,
generatekey, in order to manage the master key. A key management
extension could get the master key from an arbitrary external system.
So it also could layer an another key in between the master key and
the per-tablespace key.

Do you have any thoughts on the master key management? That design
would be flexible but complicated. Especially, the API design would be
controversial.

There is a way to enter the passphrase to Postgres to get the master
key stored in the database but the passphrase could be written in the
server log if we pass it using SQL command, which is bad. It would
require to invent another system to prevent particular SQL from being
written to the server log. Another example is to enter the password or
passphrase via a command line option. But it also could require users
to write the plain passphrase or password in script files, which is
bad too.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Jun 17, 2019 at 11:02 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote:
> >On 6/17/19 8:29 AM, Masahiko Sawada wrote:
> >> From perspective of  cryptographic, I think the fine grained TDE would
> >> be better solution. Therefore if we eventually want the fine grained
> >> TDE I wonder if it might be better to develop the table/tablespace TDE
> >> first while keeping it simple as much as possible in v1, and then we
> >> can provide the functionality to encrypt other data in database
> >> cluster to satisfy the encrypting-everything requirement. I guess that
> >> it's easier to incrementally add encryption target objects rather than
> >> making it fine grained while not changing encryption target objects.
> >>
> >> FWIW I'm writing a draft patch of per tablespace TDE and will submit
> >> it in this month. We can more discuss the complexity of the proposed
> >> TDE using it.
> >
> >+1
> >
> >Looking forward to it.
> >
>
> Yep. In particular, I'm interested in those aspects:
>

Attached the draft version patch sets of per tablespace transparent
data at rest encryption. The patch doesn't support full functionality,
it includes:

* Per tablespace encryption
* Encryption and decryption buffer data when disk I/O.
* 2 tier key hierarchy and key rotation
* Temporary file encryption (based on the patch Antonin proposd)
* System catalog encryption
* Generic key management API and test module
* Simple TAP test

but doesn't include for now (I'm writing):

* WAL encryption
* Replication supports
* pg_upgrade support
* Documentation
* README

and doesn't support:

* SLRU data encryption
* other system file encryption (pg_twophase, pg_subtrans, backup_label etc)
* Server log encryption

Before explaining the detail of the patch let me share my thoughts on
the following points.

> (1) What's the proposed minimum viable product, and how do we expect to
> extend it with the more elaborate features. I don't expect perfect
> specification, but we should have some idea so that we don't paint
> ourselves in the corner.

I think the minimum viable product should support the following features.

* Fine grained encryption object control (not using single key for
whole database cluster).
* Encrypt and decrypt tables (including system catalogs), indexes,
TOAST tables, WAL and temporary files when disk I/O.
* Passing either password, passphrase or encryption key to postgres
server without the risk of being written to files.
* Front-end programs provided by PostgreSQL source code work as much
as possible.
* Key rotation

I think that the following features would be added.

* SLRU and other data encryption. I think we can use an another
encryption key for these data.
* Support other encryption algorithms. I don't have any idea so far
but it would be not hard to support other symmetric-key algorithm.
* Faster key rotation. It can be done by having 2 tier key hierarchy.
* Integrate with external key management services. The patch
implements this but I'm sure there are other ways to integrate with
external key management services.

>
> (2) How does it affect recovery, backups and replication (both physical
> and logical)? That is, which other parts need to know the encryption keys
> to function properly?

If we encrypt whole 8kB WAL block (in cluster-wide encryption case) it
would be not hard because we just encrypt before writing to the disk
with single key. On the other hand if we encrypt some WAL records it
could be hard; it requires changes around WAL assembly code so that it
can obtain encryption keys and encrypt WAL data before inserting to
the WAL buffer. Since WAL is encrypted the recovery needs to obtain
all encryption keys and decrypt the encrypted WAL.

For streaming replication, since basically wal senders don't need to
know the actual contents of WAL (although xlogreader need to know WAL
header for validation) they send WAL data in encrypted state. And wal
receiver decrypt them. Therefore encryption keys also must be
replicated. On the other hand, logical replication (and logical
decoding) needs to decrypt WAL data when decoding. Since the logical
decoding is performed in PostgreSQL server side it's not hard to
obtain all encryption keys. It can send change sets both in
unencrypted state and even in encrypted sate if encrypt them again. We
would change xlogreader code so that it can decrypt WAL. So I think
that logical replication will be able to get WAL data in unencrypted
state without special operation.

For backups, physical backup must be encrypted even if we get it by
pg_basebackup, otherwise we cannot protect data from a malicious
backup operator threats. And encryption keys also must be backed up
together. Because this is data at rest encryption, logical backups can
be taken in unencrypted state. I think we would need nothing special
for backups.

>
> (3) What does it mean for external tools (pg_waldump, pg_upgrade,
> pg_rewind etc.)?

I think that this definitely affects at least pg_waldump, pg_upgrade,
pg_checksums and pg_rewind. By changing WAL format or giving
encryption keys to these programs we can support pg_waldump and
pg_rewind even for encrypted database. I prefer the former because
passing encryption keys to front-end programs could be risk of key
leakage. It also would affects external tools that reads or writes
database file and WAL directly. For instance pg_rman, which is a
recovery management tool, read database file and takes a backup
without a hole in each pages. Such programs would need encryption
keys.

Here is the details of patches.

Usage
======
To enable TDE feature please specify --with-openssl configuration
option. Also, please set kmgr_plugin_library GUC parameter in
postgresql.conf, which specifies the library for key managemnt
program. The patch includes contrib/kmgr_file which is the test
program for key management and store the master key in the local disk.
So for test purpose you can set kmgr_plugin_library = 'kmgr_file'.

After starting up postgres server, you can create an encrypted
tablespace by specifying 'encryption' option like,

CREATE TABLESPACE enctblspc LOCATION '/path/to/tblsp' WITH (encryption = on);

And then the tables, indexes and TOAST tables created on the
tablespace will be encrypted at rest.

For system catalogs, system catalogs on pg_default and global are not
encrypted. If you want to encrypt system catalogs, we need to create a
database on an encrypted tablespace. During copying database file from
source database we either encrypt/reencrypt each system catalogs.
You can enable and disable encryption of the table by moving
tablepsace between encrypted tablespace and non-encrypted tablespace.

Changes
=======

* 0001-Add-encryption-module-supporting-AES-256-by-using-op.patch

This patch is mostly based on the patch Antoin proposed[1] but I
modified some contents. This patch adds encrption function and
decryption function using openssl. It currently support AES-256-XTS
for buffer data encryption and AES-256-CTE for WAL encryption.

* 0002-Add-kmgr-plugin-APIs.patch

This patch adds new generic key managment APIs: startup, get,
generate, isexist and remove. Kmgr plugin programs can define these
primitive function to manage the master key that could be located at
external server. The plugin program is specified by
'kmgr_plugin_library' GUC parameter, and loaded when postmaster starts
up.

0003-Add-key-management-module-for-transparent-data-encry.patch

This patch adds key management module, which is responsible for
tablespace key management. All tablepace keys are persisted to the
file on disk, called keyring file, and loaded to the hash table on the
shared memory when postmatser starts up. The tablespace keys on the
shared memory is not encrypted state. Whenever a encrypted tablepsace
is created or dropped the keyring file is modified.

Master key identifier is used as the key for the master key. It
consists of system identifier and sequence number starting from 0 like
'pg_master_key-6707524-0000'. The sequence number is incremented
whenever key rotation.

When key rotation, we generate a new master key id in PostgreSQL core
and ask the kmgr plugin to generate new master key identified by the
new master key. And then update all tablespace keys in the keyring
file by reencrypting with the new master key.

0004-Add-facility-to-give-process-local-encryption-key.patch

This patch adds functionallity to get a process-local temporary key,
which is intended to use for temporary file encryption.

0005-Encrypt-and-decrypt-data-on-encrypted-tablespace-whe.patch

This patch support buffer encrption; encrypts and decrypt database
data when disk I/O. It adds new smgr callbacks smgrencrypta and
smgrdecrypt, and mdencrypt and mddecrypt but please note that
currently the patch supports only heap and nbtree, I'm trying to
support other access methods. Basically, when bufmgr reads buffer or
writes buffer through the shared buffer the access methods don't need
to care about the buffer encryption. However when the access methods
themselves write the buffer directly to the disk it needs to call
smgrencrypt.

0006-Encrypt-buffile.patch

This is the patch proposed Antonin. Since I've not look the detail of
this patch yet I'll look it.

0007-Make-Reorderbuffer-encrypt-spilled-out-file.patch

Same as above.

0008-Support-tablespace-encryption.patch

This patch adds 'encryption' option to tablespace.

0009-Add-kmgr-plugin-test-module-kmgr_file.patch

This patch adds a test module for kmgr plugin. It generates random
master key string and stores it to the local disk. Since this store
the master key without encryption this is for test purpose. It also
has TAP test for TDE.

Feedback and comment are very welcome.


Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Sun, Jun 16, 2019 at 03:57:46PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Sun, Jun 16, 2019 at 12:42:55PM -0400, Joe Conway wrote:
> > > On 6/16/19 9:45 AM, Bruce Momjian wrote:
> > > > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> > > >> In any case it doesn't address my first point, which is limiting the
> > > >> volume encrypted with the same key. Another valid reason is you might
> > > >> have data at varying sensitivity levels and prefer different keys be
> > > >> used for each level.
> > > > 
> > > > That seems quite complex.
> > > 
> > > How? It is no more complex than encrypting at the tablespace level
> > > already gives you - in that case you get this property for free if you
> > > care to use it.
> > 
> > All keys used to encrypt WAL data must be unlocked at all times or crash
> > recovery, PITR, and replication will not stop when it hits a locked key.
> > Given that, how much value is there in allowing a key per tablespace?
> 
> There's a few different things to discuss here, admittedly, but I don't
> think it means that there's no value in having a key per tablespace.
> 
> Ideally, a given backend would only need, and only have access to, the
> keys for the tablespaces that it is allowed to operate on.  I realize
> that's a bit farther than what we're talking about today, but hopefully
> not too much to be able to consider.

What people really want with more-granular-than-cluster encryption is
the ability to supply their passphrase key _when_ they want to access
their data, and then leave and be sure their data is secure from
decryption.  That will not be possible since the WAL will be encrypted
and any replay of it will need their passphrase key to unlock it, or the
entire system will be unrecoverable.

This is a fundamental issue, and will eventually doom any more granular
encryption approach, unless we want to use the same key for all
encrypted tablespaces, create separate WALs for each tablespace, or say
recovery of some tablespaces will fail.  I doubt any of those will be
acceptable.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
> On 6/15/19 9:28 PM, Bruce Momjian wrote:
> > There are no known non-exhaustive plaintext attacks on AES:
> > 
> >     https://crypto.stackexchange.com/questions/1512/why-is-aes-resistant-to-known-plaintext-attacks
> 
> Even that non-authoritative stackexchange thread has varying opinions.
> Surely you don't claim that limiting know plaintext as much as is
> practical is a bad idea in general.

AES is used to encrypt TLS/https, and web traffic is practically always
mostly-known plaintext.  I don't know of any cases where only part of a
webpage is encrypted by TLS to avoid encrypting known plaintext.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-05, Bruce Momjian wrote:

> What people really want with more-granular-than-cluster encryption is
> the ability to supply their passphrase key _when_ they want to access
> their data, and then leave and be sure their data is secure from
> decryption.  That will not be possible since the WAL will be encrypted
> and any replay of it will need their passphrase key to unlock it, or the
> entire system will be unrecoverable.

I'm not sure I understand why WAL replay needs the passphrase to work.
Why isn't the data saved in WAL already encrypted, which can be applied
as raw bytes to each data block, without needing to decrypt anything?
Only if somebody wants to interpret the bytes they need the passphrase,
no?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> On 2019-Jul-05, Bruce Momjian wrote:
>
> > What people really want with more-granular-than-cluster encryption is
> > the ability to supply their passphrase key _when_ they want to access
> > their data, and then leave and be sure their data is secure from
> > decryption.  That will not be possible since the WAL will be encrypted
> > and any replay of it will need their passphrase key to unlock it, or the
> > entire system will be unrecoverable.
>
> I'm not sure I understand why WAL replay needs the passphrase to work.
> Why isn't the data saved in WAL already encrypted, which can be applied
> as raw bytes to each data block, without needing to decrypt anything?
> Only if somebody wants to interpret the bytes they need the passphrase,
> no?

I had been specifically thinking of tablespaces because we might be able
to do something exactly along these lines- keep which tablespace the
data is in directly in the WAL (and not encrypted), but then have the
data itself be encrypted, and with the key for that tablespace.

Splitting the WAL by tablespace would be even nicer, of course... :)

Thanks!

Stephen

On 2019-Jul-05, Stephen Frost wrote:

> I had been specifically thinking of tablespaces because we might be able
> to do something exactly along these lines- keep which tablespace the
> data is in directly in the WAL (and not encrypted), but then have the
> data itself be encrypted, and with the key for that tablespace.

Hmm, I was imagining that the user-level data is encrypted, while the
metadata such as the containing relfilenode is not encrypted and thus
can be read by system processes such as checkpointer or WAL-apply
without needing to decrypt anything.  Maybe I'm just lacking imagination
for an attack that uses that unencrypted metadata, though.

> Splitting the WAL by tablespace would be even nicer, of course... :)

Hmm, I think you would have to synchronize the apply anyway (i.e. not
replay in one tablespace ahead of a record in another tablespace with an
earlier LSN.)  What are you thinking are the gains of doing that, anyway?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jul  5, 2019 at 03:46:28PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-05, Bruce Momjian wrote:
> 
> > What people really want with more-granular-than-cluster encryption is
> > the ability to supply their passphrase key _when_ they want to access
> > their data, and then leave and be sure their data is secure from
> > decryption.  That will not be possible since the WAL will be encrypted
> > and any replay of it will need their passphrase key to unlock it, or the
> > entire system will be unrecoverable.
> 
> I'm not sure I understand why WAL replay needs the passphrase to work.
> Why isn't the data saved in WAL already encrypted, which can be applied
> as raw bytes to each data block, without needing to decrypt anything?
> Only if somebody wants to interpret the bytes they need the passphrase,
> no?

Uh, well, you have the WAL record, and you want to write it to an 8k
page.  You have to read the 8k page from disk into shared buffers, and
you have to decrypt the 8k page to do that, right?  We aren't going to
store 8k pages encrypted in shared buffers, right?

If you did want to do that, or wanted to write them to disk without
decrypting the 8k page, it still would not work since AES is a 16-byte
encryption cipher.  I don't think we can break 8k pages up into 16-byte
chunks and be sure we can just place data into those 16-byte boundaries.

Also, that assumes that we are only encrypting the WAL payload, and not
the relation oids or other header information, which I think is a
mistake because it will lead to information leakage.

You can use AES in stream cipher mode, but then the ordering of the
encryption is critical and you can't move 16-byte chunks around --- they
have to be decrypted in their encrypted order.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 05, 2019 at 03:38:28PM -0400, Bruce Momjian wrote:
>On Sun, Jun 16, 2019 at 03:57:46PM -0400, Stephen Frost wrote:
>> Greetings,
>>
>> * Bruce Momjian (bruce@momjian.us) wrote:
>> > On Sun, Jun 16, 2019 at 12:42:55PM -0400, Joe Conway wrote:
>> > > On 6/16/19 9:45 AM, Bruce Momjian wrote:
>> > > > On Sun, Jun 16, 2019 at 07:07:20AM -0400, Joe Conway wrote:
>> > > >> In any case it doesn't address my first point, which is limiting the
>> > > >> volume encrypted with the same key. Another valid reason is you might
>> > > >> have data at varying sensitivity levels and prefer different keys be
>> > > >> used for each level.
>> > > >
>> > > > That seems quite complex.
>> > >
>> > > How? It is no more complex than encrypting at the tablespace level
>> > > already gives you - in that case you get this property for free if you
>> > > care to use it.
>> >
>> > All keys used to encrypt WAL data must be unlocked at all times or crash
>> > recovery, PITR, and replication will not stop when it hits a locked key.
>> > Given that, how much value is there in allowing a key per tablespace?
>>
>> There's a few different things to discuss here, admittedly, but I don't
>> think it means that there's no value in having a key per tablespace.
>>
>> Ideally, a given backend would only need, and only have access to, the
>> keys for the tablespaces that it is allowed to operate on.  I realize
>> that's a bit farther than what we're talking about today, but hopefully
>> not too much to be able to consider.
>
>What people really want with more-granular-than-cluster encryption is
>the ability to supply their passphrase key _when_ they want to access
>their data, and then leave and be sure their data is secure from
>decryption.  That will not be possible since the WAL will be encrypted
>and any replay of it will need their passphrase key to unlock it, or the
>entire system will be unrecoverable.
>
>This is a fundamental issue, and will eventually doom any more granular
>encryption approach, unless we want to use the same key for all
>encrypted tablespaces, create separate WALs for each tablespace, or say
>recovery of some tablespaces will fail.  I doubt any of those will be
>acceptable.
>

I agree this is a pretty crucial challenge, and those requirements seem
in direct conflict. Users use encryption to protect privacy of the data,
but we need access to some of the data to implement some of the
important tasks of a RDBMS.

And it's not just about things like recovery or replication. How do you
do ANALYZE on encrypted data? Sure, if a user runs it in a session that
has the right key, that's fine. But what about autovacuum/autoanalyze?

I suspect the issue here is that we're trying to retrofit a solution for
data-at-rest encryption to something that seems closer to protecting
data during execution.

Which is a worthwhile goal, of course, but perhaps we're trying to use
the wrong tool to achieve it? To paraphrase the hammer/nail saying "If
all you know is a block encryption, everything looks like a block."


What if the granular encryption (not the "whole cluster with a single
key") case does not encrypt whole blocks, but just tuple data? Would
that allow at least the most critical WAL use cases (recovery, physical
replication) to work without having to know all the encryption keys?

Of course, that would be a much less efficient compared to plain block
encryption, but that may be the "natural cost" of the feature.

It would not solve e.g. logical replication or ANALYZE, which both
require access to the plaintext data, though.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 2019-Jul-05, Bruce Momjian wrote:

> Uh, well, you have the WAL record, and you want to write it to an 8k
> page.  You have to read the 8k page from disk into shared buffers, and
> you have to decrypt the 8k page to do that, right?  We aren't going to
> store 8k pages encrypted in shared buffers, right?

Oh, is that the idea?  I was kinda assuming that the data was kept
as-stored in shared buffers, ie. it would be decrypted on access, not on
read from disk.  The system seems very prone to leakage if you have it
decrypted in shared memory.

If you keep it unencrypted in shared_buffers, you'd require WAL replay
and checkpointer to have every single key in the system.  That sounds
nightmarish -- a single user can create a table, hide the key and block
WAL replay and checkpointing for the whole system.

I haven't read the whole thread, sorry about that -- maybe these points
have already been discussed.

> Also, that assumes that we are only encrypting the WAL payload, and not
> the relation oids or other header information, which I think is a
> mistake because it will lead to information leakage.

Well, that was part of my question.  Why do you care to encrypt metadata
such as the relation OID (or really relfilenode)?  Yes, I was assuming
that only the WAL payload is encrypted.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jul  5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-05, Bruce Momjian wrote:
> 
> > Uh, well, you have the WAL record, and you want to write it to an 8k
> > page.  You have to read the 8k page from disk into shared buffers, and
> > you have to decrypt the 8k page to do that, right?  We aren't going to
> > store 8k pages encrypted in shared buffers, right?
> 
> Oh, is that the idea?  I was kinda assuming that the data was kept
> as-stored in shared buffers, ie. it would be decrypted on access, not on
> read from disk.  The system seems very prone to leakage if you have it
> decrypted in shared memory.

Well, the overhead of decrypting on every access will make the slowdown
huge, and I don't know what security value that would have.  I am not
sure what security value TDE itself has, but I think encrypting shared
buffer contents has even less.

> If you keep it unencrypted in shared_buffers, you'd require WAL replay
> and checkpointer to have every single key in the system.  That sounds
> nightmarish -- a single user can create a table, hide the key and block
> WAL replay and checkpointing for the whole system.

Yep, bingo!

> I haven't read the whole thread, sorry about that -- maybe these points
> have already been discussed.
> 
> > Also, that assumes that we are only encrypting the WAL payload, and not
> > the relation oids or other header information, which I think is a
> > mistake because it will lead to information leakage.
> 
> Well, that was part of my question.  Why do you care to encrypt metadata
> such as the relation OID (or really relfilenode)?  Yes, I was assuming
> that only the WAL payload is encrypted.

Well, you would need to decide what WAL information needs to be secured.
Is the fact an insert was performed on a table a security issue? 
Depends on your risks.  My point is that almost anything you do beyond
cluster-level encryption either adds complexity that is bug-prone or
fragile, or adds unacceptable overhead, or leaks security information.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul  5, 2019 at 05:00:42PM -0400, Bruce Momjian wrote:
> On Fri, Jul  5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote:
> > On 2019-Jul-05, Bruce Momjian wrote:
> > 
> > > Uh, well, you have the WAL record, and you want to write it to an 8k
> > > page.  You have to read the 8k page from disk into shared buffers, and
> > > you have to decrypt the 8k page to do that, right?  We aren't going to
> > > store 8k pages encrypted in shared buffers, right?
> > 
> > Oh, is that the idea?  I was kinda assuming that the data was kept
> > as-stored in shared buffers, ie. it would be decrypted on access, not on
> > read from disk.  The system seems very prone to leakage if you have it
> > decrypted in shared memory.
> 
> Well, the overhead of decrypting on every access will make the slowdown
> huge, and I don't know what security value that would have.  I am not
> sure what security value TDE itself has, but I think encrypting shared
> buffer contents has even less.

Sorry for the delay --- here is some benchmark info:

    https://www.postgresql.org/message-id/4723a402-b14f-4994-2de9-d85b55a56b7f%40cybertec.at

    as far as benchmarking is concerned: i did a quick test yesterday (not 
    with the final AES implementation yet) and i got pretty good results. 
    with a reasonably well cached database in a typical application I expect
    
    to loose around 10-20%. if everything fits in memory there is 0 loss of 
    course. the worst I got with the standard AES (no hardware support used 
    yet) I lost around 45% or so. but this requires a value as low as 32 MB 
    of shared buffers or so.

Notice the 0% overhead if everything fits in RAM, meaning it is not
decrypting on RAM access.  If it is 10-20% for a "reasonably well cached
database", I am sure it will be 10x that for decrypting on RAM access.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul  5, 2019 at 05:00:42PM -0400, Bruce Momjian wrote:
> On Fri, Jul  5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote:
> > On 2019-Jul-05, Bruce Momjian wrote:
> > 
> > > Uh, well, you have the WAL record, and you want to write it to an 8k
> > > page.  You have to read the 8k page from disk into shared buffers, and
> > > you have to decrypt the 8k page to do that, right?  We aren't going to
> > > store 8k pages encrypted in shared buffers, right?
> > 
> > Oh, is that the idea?  I was kinda assuming that the data was kept
> > as-stored in shared buffers, ie. it would be decrypted on access, not on
> > read from disk.  The system seems very prone to leakage if you have it
> > decrypted in shared memory.
> 
> Well, the overhead of decrypting on every access will make the slowdown
> huge, and I don't know what security value that would have.  I am not
> sure what security value TDE itself has, but I think encrypting shared
> buffer contents has even less.

Sorry I didn't answer your question directly.  Since the shared buffers
are in memory, if the decryption key is also unlocked in memory, there
isn't much value to encrypting shared buffers, and the overhead would be
huge.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul  5, 2019 at 04:10:04PM -0400, Bruce Momjian wrote:
> On Fri, Jul  5, 2019 at 03:46:28PM -0400, Alvaro Herrera wrote:
> > On 2019-Jul-05, Bruce Momjian wrote:
> > 
> > > What people really want with more-granular-than-cluster encryption is
> > > the ability to supply their passphrase key _when_ they want to access
> > > their data, and then leave and be sure their data is secure from
> > > decryption.  That will not be possible since the WAL will be encrypted
> > > and any replay of it will need their passphrase key to unlock it, or the
> > > entire system will be unrecoverable.
> > 
> > I'm not sure I understand why WAL replay needs the passphrase to work.
> > Why isn't the data saved in WAL already encrypted, which can be applied
> > as raw bytes to each data block, without needing to decrypt anything?
> > Only if somebody wants to interpret the bytes they need the passphrase,
> > no?
> 
> Uh, well, you have the WAL record, and you want to write it to an 8k
> page.  You have to read the 8k page from disk into shared buffers, and
> you have to decrypt the 8k page to do that, right?  We aren't going to
> store 8k pages encrypted in shared buffers, right?
> 
> If you did want to do that, or wanted to write them to disk without
> decrypting the 8k page, it still would not work since AES is a 16-byte
> encryption cipher.  I don't think we can break 8k pages up into 16-byte
> chunks and be sure we can just place data into those 16-byte boundaries.

I am not sure I was clear in describing this.  If you want to copy data
directly from WAL to the 8k pages, you have to use either block mode or
streaming mode for both 8k pages and WAL.  If you use block mode, then
changing any data on the pages will change all encrypted storage in the
same pages after the change, and computing WAL will require 16-byte
boundries.  If you use streaming mode, you have to compute the proper
stream at the point in the 8k pages where you are changing the row.  My
point is that in neither case can you just encrypt for the row for WAL
and assume it can be placed in an 8k pages.  Neither option seems
desirable.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-05, Bruce Momjian wrote:

> On Fri, Jul  5, 2019 at 05:00:42PM -0400, Bruce Momjian wrote:
> > On Fri, Jul  5, 2019 at 04:24:54PM -0400, Alvaro Herrera wrote:

> > > Oh, is that the idea?  I was kinda assuming that the data was kept
> > > as-stored in shared buffers, ie. it would be decrypted on access, not on
> > > read from disk.  The system seems very prone to leakage if you have it
> > > decrypted in shared memory.
> > 
> > Well, the overhead of decrypting on every access will make the slowdown
> > huge, and I don't know what security value that would have.  I am not
> > sure what security value TDE itself has, but I think encrypting shared
> > buffer contents has even less.
> 
> Sorry I didn't answer your question directly.  Since the shared buffers
> are in memory, if the decryption key is also unlocked in memory, there
> isn't much value to encrypting shared buffers, and the overhead would be
> huge.

Oh, I get your point now.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jul  5, 2019 at 10:24:39PM +0200, Tomas Vondra wrote:
> I agree this is a pretty crucial challenge, and those requirements seem
> in direct conflict. Users use encryption to protect privacy of the data,
> but we need access to some of the data to implement some of the
> important tasks of a RDBMS.
> 
> And it's not just about things like recovery or replication. How do you
> do ANALYZE on encrypted data? Sure, if a user runs it in a session that
> has the right key, that's fine. But what about autovacuum/autoanalyze?

There might be a way to defer ANALYZE and autovacuum/autoanalyze, but
what about VACUUM FREEZE?  We can defer that too, but not the clog
truncation that is eventually the product of the freeze.

What about referential integrity constraints that need to check primary
keys in the encrypted tables?  I also don't see a way of delaying that,
and if you can't do referential integrity into the encrypted tables, it
reduces the value of having encrypted data in the same database rather
than in another database or cluster?

I still feel we have not clearly described what the options are:

1.  Encrypt everything

2.  Encrypt only some tables (for performance reasons), and use only one
key, or use multiple keys to allow for key rotation.  All keys are
always unlocked.

3.  Encrypt only some tables with different keys, and the keys are not
always unlocked.

As Tomas already stated, using tablespaces to distinguish encrypted from
non-encrypted tables doesn't make sense since the file system used for
the storage is immaterial to the encryptions status. An easier way would
be to just add a bit to WAL that would indicate if the rest of the WAL
record is encrypted, though I doubt the performance boost is worth the
complexity.

I see the attraction of #3, but operationally it is unclear how we can
decouple data that is not always accessible, as outlined above.  We
could probably work around the WAL issues, but it is going to be much
more overhead.  It is also unclear how the user supplies the keys ---
are they done at boot time, and if so, how are later keys unlocked, or
does the client provide it?  If the client provides it, isn't it better
to do client-side encryption, or have the client use pgcrypto with some
key management around it like pgcryptokey?  This presentation shows how
to use triggers to implement transparent encryption at the column level:

    https://momjian.us/main/writings/crypto_hw_use.pdf#page=77

Structurally, I understand the desire to push key control out to users
in #3, but it then becomes very complex to construct a system where the
data is tightly coupled in a Postgres cluster.  The pgcrypto method
above works because it decouples row control, like xmin/xmax and WAL
replay, which is not encrypted, with the row payload, which is
encrypted.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-07-05 22:24, Tomas Vondra wrote:
> What if the granular encryption (not the "whole cluster with a single
> key") case does not encrypt whole blocks, but just tuple data? Would
> that allow at least the most critical WAL use cases (recovery, physical
> replication) to work without having to know all the encryption keys?

Finding the exact point where you divide up sensitive and non-sensitive
data would be difficult.

For example, say, you encrypt the tuple payload but not the tuple
header, so that vacuum would still work.  Then, someone who has access
to the raw data directory could infer in combination with commit
timestamps for example, that on Friday between 5pm and 6pm, 10000
records were updated, 500 were inserted, and 200 were deleted, and that
table has about this size, and this happens every Friday, and so on.
That seems way to much information to reveal for an allegedly encrypted
data directory.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sun, Jul 7, 2019 at 1:05 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Fri, Jul  5, 2019 at 10:24:39PM +0200, Tomas Vondra wrote:
> > I agree this is a pretty crucial challenge, and those requirements seem
> > in direct conflict. Users use encryption to protect privacy of the data,
> > but we need access to some of the data to implement some of the
> > important tasks of a RDBMS.
> >
> > And it's not just about things like recovery or replication. How do you
> > do ANALYZE on encrypted data? Sure, if a user runs it in a session that
> > has the right key, that's fine. But what about autovacuum/autoanalyze?
>
> There might be a way to defer ANALYZE and autovacuum/autoanalyze, but
> what about VACUUM FREEZE?  We can defer that too, but not the clog
> truncation that is eventually the product of the freeze.
>
> What about referential integrity constraints that need to check primary
> keys in the encrypted tables?  I also don't see a way of delaying that,
> and if you can't do referential integrity into the encrypted tables, it
> reduces the value of having encrypted data in the same database rather
> than in another database or cluster?
>

I just thought that PostgreSQL's auxiliary processes such as
autovacuum, startup, checkpointer, bgwriter should always be able to
access all keys because there are already in inside database. Even
today these processes don't check any privileges when accessing to
data. What security threats we can protect data from by requiring
privileges even for auxiliary processes? If this is a security problem
isn't it also true for cluster-wide encryption? I guess that processes
who have an access privilege on the table can always get the
corresponding encryption key. And any processes cannot access an
encryption key directly without accessing to a database object.

> I still feel we have not clearly described what the options are:
>
> 1.  Encrypt everything
>
> 2.  Encrypt only some tables (for performance reasons), and use only one
> key, or use multiple keys to allow for key rotation.  All keys are
> always unlocked.
>
> 3.  Encrypt only some tables with different keys, and the keys are not
> always unlocked.
>
> As Tomas already stated, using tablespaces to distinguish encrypted from
> non-encrypted tables doesn't make sense since the file system used for
> the storage is immaterial to the encryptions status. An easier way would
> be to just add a bit to WAL that would indicate if the rest of the WAL
> record is encrypted, though I doubt the performance boost is worth the
> complexity.

Okay, instead of using tablespaces we can create groups grouping
tables being encrypted with the same key. I think the one of the most
important point here is to provide a granular encryption feature and
have less the number of keys in database cluster, not to provide per
tablespace encryption feature. I'm not going to insist it should be
per tablespace encryption.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Jul  8, 2019 at 06:04:28PM +0900, Masahiko Sawada wrote:
> On Sun, Jul 7, 2019 at 1:05 AM Bruce Momjian <bruce@momjian.us> wrote:
> > What about referential integrity constraints that need to check primary
> > keys in the encrypted tables?  I also don't see a way of delaying that,
> > and if you can't do referential integrity into the encrypted tables, it
> > reduces the value of having encrypted data in the same database rather
> > than in another database or cluster?
> 
> I just thought that PostgreSQL's auxiliary processes such as
> autovacuum, startup, checkpointer, bgwriter should always be able to
> access all keys because there are already in inside database. Even
> today these processes don't check any privileges when accessing to
> data. What security threats we can protect data from by requiring
> privileges even for auxiliary processes? If this is a security problem
> isn't it also true for cluster-wide encryption? I guess that processes
> who have an access privilege on the table can always get the
> corresponding encryption key. And any processes cannot access an
> encryption key directly without accessing to a database object.

Well, see my list of three things that users want in an earlier email:

    https://www.postgresql.org/message-id/20190706160514.b67q4f7abcxfdahk@momjian.us

When people are asking for multiple keys (not just for key rotation),
they are asking to have multiple keys that can be supplied by users only
when they need to access the data.  Yes, the keys are always in the
datbase, but the feature request is that they are only unlocked when the
user needs to access the data.  Obviously, that will not work for
autovacuum when the encryption is at the block level.

If the key is always unlocked, there is questionable security value of
having multiple keys, beyond key rotation.

> > I still feel we have not clearly described what the options are:
> >
> > 1.  Encrypt everything
> >
> > 2.  Encrypt only some tables (for performance reasons), and use only one
> > key, or use multiple keys to allow for key rotation.  All keys are
> > always unlocked.
> >
> > 3.  Encrypt only some tables with different keys, and the keys are not
> > always unlocked.
> >
> > As Tomas already stated, using tablespaces to distinguish encrypted from
> > non-encrypted tables doesn't make sense since the file system used for
> > the storage is immaterial to the encryptions status. An easier way would
> > be to just add a bit to WAL that would indicate if the rest of the WAL
> > record is encrypted, though I doubt the performance boost is worth the
> > complexity.
> 
> Okay, instead of using tablespaces we can create groups grouping
> tables being encrypted with the same key. I think the one of the most
> important point here is to provide a granular encryption feature and

Why is this important?  What are you trying to accomplish?

> have less the number of keys in database cluster, not to provide per
> tablespace encryption feature. I'm not going to insist it should be
> per tablespace encryption.

It is unclear which item you are looking for.  Which number are you
suggesting from the three listed above in the email URL?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 7/8/19 10:19 AM, Bruce Momjian wrote:
> When people are asking for multiple keys (not just for key rotation),
> they are asking to have multiple keys that can be supplied by users only
> when they need to access the data.  Yes, the keys are always in the
> datbase, but the feature request is that they are only unlocked when the
> user needs to access the data.  Obviously, that will not work for
> autovacuum when the encryption is at the block level.

> If the key is always unlocked, there is questionable security value of
> having multiple keys, beyond key rotation.

That is not true. Having multiple keys also allows you to reduce the
amount of data encrypted with a single key, which is desirable because:

1. It makes cryptanalysis more difficult
2. Puts less data at risk if someone gets "lucky" in doing brute force


Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
> On 7/8/19 10:19 AM, Bruce Momjian wrote:
> > When people are asking for multiple keys (not just for key rotation),
> > they are asking to have multiple keys that can be supplied by users only
> > when they need to access the data.  Yes, the keys are always in the
> > datbase, but the feature request is that they are only unlocked when the
> > user needs to access the data.  Obviously, that will not work for
> > autovacuum when the encryption is at the block level.
> 
> > If the key is always unlocked, there is questionable security value of
> > having multiple keys, beyond key rotation.
> 
> That is not true. Having multiple keys also allows you to reduce the
> amount of data encrypted with a single key, which is desirable because:
> 
> 1. It makes cryptanalysis more difficult
> 2. Puts less data at risk if someone gets "lucky" in doing brute force

What systems use multiple keys like that?  I know of no website that
does that.  Your arguments seem hypothetical.  What is your goal here?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
> > On 7/8/19 10:19 AM, Bruce Momjian wrote:
> > > When people are asking for multiple keys (not just for key rotation),
> > > they are asking to have multiple keys that can be supplied by users only
> > > when they need to access the data.  Yes, the keys are always in the
> > > datbase, but the feature request is that they are only unlocked when the
> > > user needs to access the data.  Obviously, that will not work for
> > > autovacuum when the encryption is at the block level.
> >
> > > If the key is always unlocked, there is questionable security value of
> > > having multiple keys, beyond key rotation.
> >
> > That is not true. Having multiple keys also allows you to reduce the
> > amount of data encrypted with a single key, which is desirable because:
> >
> > 1. It makes cryptanalysis more difficult
> > 2. Puts less data at risk if someone gets "lucky" in doing brute force
>
> What systems use multiple keys like that?  I know of no website that
> does that.  Your arguments seem hypothetical.  What is your goal here?

Not sure what the reference to 'website' is here, but one doesn't get
certificates for TLS/SSL usage that aren't time-bounded, and when it
comes to the actual on-the-wire encryption that's used, that's a
symmetric key that's generated on-the-fly for every connection.

Wouldn't the fact that they generate a different key for every
connection be a pretty clear indication that it's a good idea to use
multiple keys and not use the same key over and over..?

Of course, we can discuss if what websites do with over-the-wire
encryption is sensible to compare to what we want to do in PG for
data-at-rest, but then we shouldn't be talking about what websites do,
it'd make more sense to look at other data-at-rest encryption systems
and consider what they're doing.

Thanks,

Stephen

On 2019-07-08 17:47, Stephen Frost wrote:
> Of course, we can discuss if what websites do with over-the-wire
> encryption is sensible to compare to what we want to do in PG for
> data-at-rest, but then we shouldn't be talking about what websites do,
> it'd make more sense to look at other data-at-rest encryption systems
> and consider what they're doing.

So, how do encrypted file systems do it?  Are there any encrypted file
systems in general use that allow encrypting only some files or
encrypting different parts of the file system with different keys, or
any of those other granular approaches being discussed?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7/8/19 11:56 AM, Peter Eisentraut wrote:
> On 2019-07-08 17:47, Stephen Frost wrote:
>> Of course, we can discuss if what websites do with over-the-wire
>> encryption is sensible to compare to what we want to do in PG for
>> data-at-rest, but then we shouldn't be talking about what websites do,
>> it'd make more sense to look at other data-at-rest encryption systems
>> and consider what they're doing.
>
> So, how do encrypted file systems do it?  Are there any encrypted file
> systems in general use that allow encrypting only some files or
> encrypting different parts of the file system with different keys, or
> any of those other granular approaches being discussed?

Well it is fairly common, for good reason IMHO, to encrypt some mount
points and not others on a system. In my mind, and in practice to a
large extent, a postgres tablespace == a unique mount point.

There is a description here:

  https://wiki.archlinux.org/index.php/Disk_encryption

A pertinent quote:
----
After it has been derived, the master key is securely stored in memory
(e.g. in a kernel keyring), for as long as the encrypted block device or
folder is mounted.

It is usually not used for de/encrypting the disk data directly, though.
For example, in the case of stacked filesystem encryption, each file can
be automatically assigned its own encryption key. Whenever the file is
to be read/modified, this file key first needs to be decrypted using the
main key, before it can itself be used to de/encrypt the file contents:

                           ╭┈┈┈┈┈┈┈┈┈┈┈┈╮
                           ┊ master key ┊
   file on disk:           ╰┈┈┈┈┈┬┈┈┈┈┈┈╯
  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐        │
  ╎╭───────────────────╮╎        ▼          ╭┈┈┈┈┈┈┈┈┈┈╮
  ╎│ encrypted file key│━━━━(decryption)━━━▶┊ file key ┊
  ╎╰───────────────────╯╎                   ╰┈┈┈┈┬┈┈┈┈┈╯
  ╎┌───────────────────┐╎                        ▼
┌┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┐
  ╎│ encrypted file    │◀━━━━━━━━━━━━━━━━━(de/encryption)━━━▶┊ readable
file ┊
  ╎│ contents          │╎                                    ┊ contents
     ┊
  ╎└───────────────────┘╎
└┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┘
  └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

In a similar manner, a separate key (e.g. one per folder) may be used
for the encryption of file names in the case of stacked filesystem
encryption.
----

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Mon, Jul  8, 2019 at 11:47:33AM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
> > > On 7/8/19 10:19 AM, Bruce Momjian wrote:
> > > > When people are asking for multiple keys (not just for key rotation),
> > > > they are asking to have multiple keys that can be supplied by users only
> > > > when they need to access the data.  Yes, the keys are always in the
> > > > datbase, but the feature request is that they are only unlocked when the
> > > > user needs to access the data.  Obviously, that will not work for
> > > > autovacuum when the encryption is at the block level.
> > > 
> > > > If the key is always unlocked, there is questionable security value of
> > > > having multiple keys, beyond key rotation.
> > > 
> > > That is not true. Having multiple keys also allows you to reduce the
> > > amount of data encrypted with a single key, which is desirable because:
> > > 
> > > 1. It makes cryptanalysis more difficult
> > > 2. Puts less data at risk if someone gets "lucky" in doing brute force
> > 
> > What systems use multiple keys like that?  I know of no website that
> > does that.  Your arguments seem hypothetical.  What is your goal here?
> 
> Not sure what the reference to 'website' is here, but one doesn't get
> certificates for TLS/SSL usage that aren't time-bounded, and when it
> comes to the actual on-the-wire encryption that's used, that's a
> symmetric key that's generated on-the-fly for every connection.
> 
> Wouldn't the fact that they generate a different key for every
> connection be a pretty clear indication that it's a good idea to use
> multiple keys and not use the same key over and over..?
> 
> Of course, we can discuss if what websites do with over-the-wire
> encryption is sensible to compare to what we want to do in PG for
> data-at-rest, but then we shouldn't be talking about what websites do,
> it'd make more sense to look at other data-at-rest encryption systems
> and consider what they're doing.

(I talked to Joe on chat for clarity.)  In modern TLS, the certificate is
used only for authentication, and Diffie–Hellman is used for key
exchange:

    https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange

So, the question is whether you can pass so much data in TLS that using
the same key for the entire session is a security issue.  TLS originally
had key renegotiation, but that was removed in TLS 1.3:

    https://www.cloudinsidr.com/content/known-attack-vectors-against-tls-implementation-vulnerabilities/
    To mitigate these types of attacks, TLS 1.3 disallows renegotiation.

Of course, a database is going to process even more data so if the
amount of data encrypted is a problem, we might have a problem too in
using a single key.  This is not related to whether we use one key for
the entire cluster or multiple keys per tablespace --- the problem is
the same.  I guess we could create 1024 keys and use the bottom bits of
the block number to decide what key to use.  However, that still only
pushes the goalposts farther.

Anyway, I will to research the reasonable data size that can be secured
with a single key via AES.  I will look at how PGP encrypts large files
too.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul 08, 2019 at 11:25:10AM -0400, Bruce Momjian wrote:
>On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
>> On 7/8/19 10:19 AM, Bruce Momjian wrote:
>> > When people are asking for multiple keys (not just for key rotation),
>> > they are asking to have multiple keys that can be supplied by users only
>> > when they need to access the data.  Yes, the keys are always in the
>> > datbase, but the feature request is that they are only unlocked when the
>> > user needs to access the data.  Obviously, that will not work for
>> > autovacuum when the encryption is at the block level.
>>
>> > If the key is always unlocked, there is questionable security value of
>> > having multiple keys, beyond key rotation.
>>
>> That is not true. Having multiple keys also allows you to reduce the
>> amount of data encrypted with a single key, which is desirable because:
>>
>> 1. It makes cryptanalysis more difficult
>> 2. Puts less data at risk if someone gets "lucky" in doing brute force
>
>What systems use multiple keys like that?  I know of no website that
>does that.  Your arguments seem hypothetical.  What is your goal here?
>

I might ask the same question about databases - which databases use an
encryption scheme where the database does not have access to the keys?

Actually, I've already asked this question before ...

The databases I'm familiar with do store all the keys in a vault that's
unlocked during startup, and then users may get keys from it (including
maintenance processes). We could still control access to those keys in
various ways (ACL or whatever), of course.

BTW how do you know this is what users want? Maybe they do, but then
again - maybe they just see it as magic and don't realize the extra
complexity (not just at the database level). In my experience users
generally want more abstract things, like "Ensure data privacy in case
media theft," or "protection against evil DBA".


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 11:47:33AM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
> > > > On 7/8/19 10:19 AM, Bruce Momjian wrote:
> > > > > When people are asking for multiple keys (not just for key rotation),
> > > > > they are asking to have multiple keys that can be supplied by users only
> > > > > when they need to access the data.  Yes, the keys are always in the
> > > > > datbase, but the feature request is that they are only unlocked when the
> > > > > user needs to access the data.  Obviously, that will not work for
> > > > > autovacuum when the encryption is at the block level.
> > > >
> > > > > If the key is always unlocked, there is questionable security value of
> > > > > having multiple keys, beyond key rotation.
> > > >
> > > > That is not true. Having multiple keys also allows you to reduce the
> > > > amount of data encrypted with a single key, which is desirable because:
> > > >
> > > > 1. It makes cryptanalysis more difficult
> > > > 2. Puts less data at risk if someone gets "lucky" in doing brute force
> > >
> > > What systems use multiple keys like that?  I know of no website that
> > > does that.  Your arguments seem hypothetical.  What is your goal here?
> >
> > Not sure what the reference to 'website' is here, but one doesn't get
> > certificates for TLS/SSL usage that aren't time-bounded, and when it
> > comes to the actual on-the-wire encryption that's used, that's a
> > symmetric key that's generated on-the-fly for every connection.
> >
> > Wouldn't the fact that they generate a different key for every
> > connection be a pretty clear indication that it's a good idea to use
> > multiple keys and not use the same key over and over..?
> >
> > Of course, we can discuss if what websites do with over-the-wire
> > encryption is sensible to compare to what we want to do in PG for
> > data-at-rest, but then we shouldn't be talking about what websites do,
> > it'd make more sense to look at other data-at-rest encryption systems
> > and consider what they're doing.
>
> (I talked to Joe on chat for clarity.)  In modern TLS, the certificate is
> used only for authentication, and Diffie–Hellman is used for key
> exchange:
>
>     https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange

Right, and the key that's figured out for each connection is at least
specific to the server AND client keys/certificates, thus meaning that
they're changed at least as frequently as those change (and clients end
up creating ones on the fly randomly if they don't have one, iirc).

> So, the question is whether you can pass so much data in TLS that using
> the same key for the entire session is a security issue.  TLS originally
> had key renegotiation, but that was removed in TLS 1.3:
>
>     https://www.cloudinsidr.com/content/known-attack-vectors-against-tls-implementation-vulnerabilities/
>     To mitigate these types of attacks, TLS 1.3 disallows renegotiation.

It was removed due to attacks targeting the renegotiation, not because
doing re-keying by itself was a bad idea, or because using multiple keys
was seen as a bad idea.

> Of course, a database is going to process even more data so if the
> amount of data encrypted is a problem, we might have a problem too in
> using a single key.  This is not related to whether we use one key for
> the entire cluster or multiple keys per tablespace --- the problem is
> the same.  I guess we could create 1024 keys and use the bottom bits of
> the block number to decide what key to use.  However, that still only
> pushes the goalposts farther.

All of this is about pushing the goalposts farther away, as I see it.
There's going to be trade-offs here and there isn't going to be any "one
right answer" when it comes to this space.  That's why I'm inclined to
argue that we should try to come up with a relatively *good* solution
that doesn't create a huge amount of work for us, and then build on
that.  To that end, leveraging metadata that we already have outside of
the catalogs (databases, tablespaces, potentially other information that
we store, essentially, in the filesystem metadata already) to decide on
what key to use, and how many we can support, strikes me as a good
initial target.

> Anyway, I will to research the reasonable data size that can be secured
> with a single key via AES.  I will look at how PGP encrypts large files
> too.

This seems unlikely to lead to a definitive result, but it would be
interesting to hear if there have been studies around that and what
their conclusions were.

When it comes to concerns about autovacuum or other system processes,
those don't have any direct user connections or interactions, so having
them be more privileged and having access to more is reasonable.

Ideally, all of this would leverage a vaulting system or other mechanism
which manages access to the keys and allows their usage to be limited.
That's been generally accepted as a good way to bridge the gap between
having to ask users every time for a key and having keys stored
long-term in memory.  Having *only* the keys for the data which the
currently connected user is allowed to access would certainly be a great
initial capability, even if system processes (including potentially WAL
replay) have to have access to all of the keys.  And yes, shared buffers
being unencrypted and accessible by every backend continues to be an
issue- it'd be great to improve on that situation too.  I don't think
having everything encrypted in shared buffers is likely the solution,
rather, segregating it up might make more sense, again, along similar
lines to keys and using metadata that's outside of the catalogs, which
has been discussed previously, though I don't think anyone's actively
working on it.

Thanks,

Stephen

On Mon, Jul 08, 2019 at 02:39:44PM -0400, Stephen Frost wrote:
>Greetings,
>
>* Bruce Momjian (bruce@momjian.us) wrote:
>> On Mon, Jul  8, 2019 at 11:47:33AM -0400, Stephen Frost wrote:
>> > * Bruce Momjian (bruce@momjian.us) wrote:
>> > > On Mon, Jul  8, 2019 at 11:18:01AM -0400, Joe Conway wrote:
>> > > > On 7/8/19 10:19 AM, Bruce Momjian wrote:
>> > > > > When people are asking for multiple keys (not just for key rotation),
>> > > > > they are asking to have multiple keys that can be supplied by users only
>> > > > > when they need to access the data.  Yes, the keys are always in the
>> > > > > datbase, but the feature request is that they are only unlocked when the
>> > > > > user needs to access the data.  Obviously, that will not work for
>> > > > > autovacuum when the encryption is at the block level.
>> > > >
>> > > > > If the key is always unlocked, there is questionable security value of
>> > > > > having multiple keys, beyond key rotation.
>> > > >
>> > > > That is not true. Having multiple keys also allows you to reduce the
>> > > > amount of data encrypted with a single key, which is desirable because:
>> > > >
>> > > > 1. It makes cryptanalysis more difficult
>> > > > 2. Puts less data at risk if someone gets "lucky" in doing brute force
>> > >
>> > > What systems use multiple keys like that?  I know of no website that
>> > > does that.  Your arguments seem hypothetical.  What is your goal here?
>> >
>> > Not sure what the reference to 'website' is here, but one doesn't get
>> > certificates for TLS/SSL usage that aren't time-bounded, and when it
>> > comes to the actual on-the-wire encryption that's used, that's a
>> > symmetric key that's generated on-the-fly for every connection.
>> >
>> > Wouldn't the fact that they generate a different key for every
>> > connection be a pretty clear indication that it's a good idea to use
>> > multiple keys and not use the same key over and over..?
>> >
>> > Of course, we can discuss if what websites do with over-the-wire
>> > encryption is sensible to compare to what we want to do in PG for
>> > data-at-rest, but then we shouldn't be talking about what websites do,
>> > it'd make more sense to look at other data-at-rest encryption systems
>> > and consider what they're doing.
>>
>> (I talked to Joe on chat for clarity.)  In modern TLS, the certificate is
>> used only for authentication, and Diffie–Hellman is used for key
>> exchange:
>>
>>     https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange
>
>Right, and the key that's figured out for each connection is at least
>specific to the server AND client keys/certificates, thus meaning that
>they're changed at least as frequently as those change (and clients end
>up creating ones on the fly randomly if they don't have one, iirc).
>
>> So, the question is whether you can pass so much data in TLS that using
>> the same key for the entire session is a security issue.  TLS originally
>> had key renegotiation, but that was removed in TLS 1.3:
>>
>>     https://www.cloudinsidr.com/content/known-attack-vectors-against-tls-implementation-vulnerabilities/
>>     To mitigate these types of attacks, TLS 1.3 disallows renegotiation.
>
>It was removed due to attacks targeting the renegotiation, not because
>doing re-keying by itself was a bad idea, or because using multiple keys
>was seen as a bad idea.
>
>> Of course, a database is going to process even more data so if the
>> amount of data encrypted is a problem, we might have a problem too in
>> using a single key.  This is not related to whether we use one key for
>> the entire cluster or multiple keys per tablespace --- the problem is
>> the same.  I guess we could create 1024 keys and use the bottom bits of
>> the block number to decide what key to use.  However, that still only
>> pushes the goalposts farther.
>
>All of this is about pushing the goalposts farther away, as I see it.
>There's going to be trade-offs here and there isn't going to be any "one
>right answer" when it comes to this space.  That's why I'm inclined to
>argue that we should try to come up with a relatively *good* solution
>that doesn't create a huge amount of work for us, and then build on
>that.  To that end, leveraging metadata that we already have outside of
>the catalogs (databases, tablespaces, potentially other information that
>we store, essentially, in the filesystem metadata already) to decide on
>what key to use, and how many we can support, strikes me as a good
>initial target.
>
>> Anyway, I will to research the reasonable data size that can be secured
>> with a single key via AES.  I will look at how PGP encrypts large files
>> too.
>
>This seems unlikely to lead to a definitive result, but it would be
>interesting to hear if there have been studies around that and what
>their conclusions were.
>
>When it comes to concerns about autovacuum or other system processes,
>those don't have any direct user connections or interactions, so having
>them be more privileged and having access to more is reasonable.
>

I think Bruce's proposal was to minimize the time the key is "unlocked"
in memory by only unlocking them when the user connects and supplies
some sort of secret (passphrase), and remove them from memory when the
user disconnects. So there's no way for the auxiliary processes to gain
access to those keys, because only the user knows the secret.

FWIW I have doubts this scheme actually measurably improves privacy in
practice, because most busy applications will end up having the keys in
the memory all the time anyway.

It also assumes memory is unsafe, i.e. bad actors can read it, and
that's probably a valid concern (root access, vulnerabilities etc.). But
in that case we already have plenty of issues with data in flight
anyway, and I doubt TDE is an answer to that.

>Ideally, all of this would leverage a vaulting system or other mechanism
>which manages access to the keys and allows their usage to be limited.
>That's been generally accepted as a good way to bridge the gap between
>having to ask users every time for a key and having keys stored
>long-term in memory.

Right. I agree with this.

>Having *only* the keys for the data which the
>currently connected user is allowed to access would certainly be a great
>initial capability, even if system processes (including potentially WAL
>replay) have to have access to all of the keys.  And yes, shared buffers
>being unencrypted and accessible by every backend continues to be an
>issue- it'd be great to improve on that situation too.  I don't think
>having everything encrypted in shared buffers is likely the solution,
>rather, segregating it up might make more sense, again, along similar
>lines to keys and using metadata that's outside of the catalogs, which
>has been discussed previously, though I don't think anyone's actively
>working on it.
>

I very much doubt TDE is a solution to this. Essentially, TDE is a good
data-at-rest solution, but this seems more like protecting data during
execution. And in that case I think we may need an entirely different
encryption scheme.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 08, 2019 at 12:16:04PM -0400, Bruce Momjian wrote:
>
> ...
>
>Anyway, I will to research the reasonable data size that can be secured
>with a single key via AES.  I will look at how PGP encrypts large files
>too.
>

IMO there are various recommendations about this, for example from NIST.
But it varies on the exact encryption mode (say, GCM, XTS, ...) and the
recommendations are not "per key" but "per key + nonce" etc.

IANAC but my understanding is if we use e.g. "OID + blocknum" as nonce,
then we should be pretty safe.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 08, 2019 at 12:09:58PM -0400, Joe Conway wrote:
>On 7/8/19 11:56 AM, Peter Eisentraut wrote:
>> On 2019-07-08 17:47, Stephen Frost wrote:
>>> Of course, we can discuss if what websites do with over-the-wire
>>> encryption is sensible to compare to what we want to do in PG for
>>> data-at-rest, but then we shouldn't be talking about what websites do,
>>> it'd make more sense to look at other data-at-rest encryption systems
>>> and consider what they're doing.
>>
>> So, how do encrypted file systems do it?  Are there any encrypted file
>> systems in general use that allow encrypting only some files or
>> encrypting different parts of the file system with different keys, or
>> any of those other granular approaches being discussed?
>
>Well it is fairly common, for good reason IMHO, to encrypt some mount
>points and not others on a system. In my mind, and in practice to a
>large extent, a postgres tablespace == a unique mount point.
>
>There is a description here:
>
>  https://wiki.archlinux.org/index.php/Disk_encryption
>

That link is a bit overwhelming, as it explains how various encrypted
filesystems do things. There's now official support for this in the
Linux kernel (encryption at the filesystem level, not block device) in
the form of fscrypt, see

  https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html

It's a bit different because that's not a stacked encryption, it's
integrated directly into filesystems (like ext4, at the moment) and it
leverages other kernel facilities (like keyring).

The link also discusses the threat model, which is interesting
particularly interesting for this discussion, IMO.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul  8, 2019 at 12:09:58PM -0400, Joe Conway wrote:
> On 7/8/19 11:56 AM, Peter Eisentraut wrote:
> > On 2019-07-08 17:47, Stephen Frost wrote:
> >> Of course, we can discuss if what websites do with over-the-wire
> >> encryption is sensible to compare to what we want to do in PG for
> >> data-at-rest, but then we shouldn't be talking about what websites do,
> >> it'd make more sense to look at other data-at-rest encryption systems
> >> and consider what they're doing.
> > 
> > So, how do encrypted file systems do it?  Are there any encrypted file
> > systems in general use that allow encrypting only some files or
> > encrypting different parts of the file system with different keys, or
> > any of those other granular approaches being discussed?
> 
> Well it is fairly common, for good reason IMHO, to encrypt some mount
> points and not others on a system. In my mind, and in practice to a
> large extent, a postgres tablespace == a unique mount point.

Yes, that is a normal partition point for key use because one file
system is independent of others.  You could use different keys for
different directories in the same file system, but internally it all
uses the same storage, and storage theft would potentially happen at the
file system level.

For Postgres, tablespaces are not independent of the database system,
though media theft would still match.  Of course, in the case of a
tablespace media theft, Postgres would be quite confused, though you
could still start the database system:

    SELECT * FROM test;
    ERROR:  could not open file
    "pg_tblspc/16385/PG_13_201907054/16384/16386": No such file or directory

but the data would be gone.  What you can't do with Postgres is to have
the tablespace be inaccessible and then later reappear.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul  8, 2019 at 02:39:44PM -0400, Stephen Frost wrote:
> > > Of course, we can discuss if what websites do with over-the-wire
> > > encryption is sensible to compare to what we want to do in PG for
> > > data-at-rest, but then we shouldn't be talking about what websites do,
> > > it'd make more sense to look at other data-at-rest encryption systems
> > > and consider what they're doing.
> > 
> > (I talked to Joe on chat for clarity.)  In modern TLS, the certificate is
> > used only for authentication, and Diffie–Hellman is used for key
> > exchange:
> > 
> >     https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange
> 
> Right, and the key that's figured out for each connection is at least
> specific to the server AND client keys/certificates, thus meaning that
> they're changed at least as frequently as those change (and clients end
> up creating ones on the fly randomly if they don't have one, iirc).
> 
> > So, the question is whether you can pass so much data in TLS that using
> > the same key for the entire session is a security issue.  TLS originally
> > had key renegotiation, but that was removed in TLS 1.3:
> > 
> >     https://www.cloudinsidr.com/content/known-attack-vectors-against-tls-implementation-vulnerabilities/
> >     To mitigate these types of attacks, TLS 1.3 disallows renegotiation.
> 
> It was removed due to attacks targeting the renegotiation, not because
> doing re-keying by itself was a bad idea, or because using multiple keys
> was seen as a bad idea.

Well, if it was a necessary features, I assume TLS 1.3 would have found
a way to make it secure, no?  Certainly they are not shipping TLS 1.3
with a known weakness.

> > Of course, a database is going to process even more data so if the
> > amount of data encrypted is a problem, we might have a problem too in
> > using a single key.  This is not related to whether we use one key for
> > the entire cluster or multiple keys per tablespace --- the problem is
> > the same.  I guess we could create 1024 keys and use the bottom bits of
> > the block number to decide what key to use.  However, that still only
> > pushes the goalposts farther.
> 
> All of this is about pushing the goalposts farther away, as I see it.
> There's going to be trade-offs here and there isn't going to be any "one
> right answer" when it comes to this space.  That's why I'm inclined to
> argue that we should try to come up with a relatively *good* solution
> that doesn't create a huge amount of work for us, and then build on
> that.  To that end, leveraging metadata that we already have outside of
> the catalogs (databases, tablespaces, potentially other information that
> we store, essentially, in the filesystem metadata already) to decide on
> what key to use, and how many we can support, strikes me as a good
> initial target.

Yes, we will need that for a usable nonce that we don't need to store in
the blocks and WAL files.

> > Anyway, I will to research the reasonable data size that can be secured
> > with a single key via AES.  I will look at how PGP encrypts large files
> > too.
> 
> This seems unlikely to lead to a definitive result, but it would be
> interesting to hear if there have been studies around that and what
> their conclusions were.

I found this:


https://crypto.stackexchange.com/questions/44113/what-is-a-safe-maximum-message-size-limit-when-encrypting-files-to-disk-with-aes
    https://crypto.stackexchange.com/questions/20333/encryption-of-big-files-in-java-with-aes-gcm/20340#20340

The numbers listed are:

    Maximum Encrypted Plaintext Size:  68GB
    Maximum Processed Additional Authenticated Data: 2 x 10^18

The 68GB value is "the maximum bits that can be processed with a single
key/IV(nonce) pair."  We would 8k of data for each 8k page.  If we
assume a unique nonce per page that is 10^32 bytes.

For the WAL we would probably use a different nonce for each 16MB page,
so we would be OK there too, since that is 10 ^ 36 bytes.

gives us 10^36 bytes before the segment number causes the nonce to
repeat.

> When it comes to concerns about autovacuum or other system processes,
> those don't have any direct user connections or interactions, so having
> them be more privileged and having access to more is reasonable.

Well, I am trying to understand the value of having some keys accessible
by some parts of the system, and some not.  I am unclear what security
value that has.

> Ideally, all of this would leverage a vaulting system or other mechanism
> which manages access to the keys and allows their usage to be limited.
> That's been generally accepted as a good way to bridge the gap between
> having to ask users every time for a key and having keys stored
> long-term in memory.  Having *only* the keys for the data which the
> currently connected user is allowed to access would certainly be a great
> initial capability, even if system processes (including potentially WAL
> replay) have to have access to all of the keys.  And yes, shared buffers
> being unencrypted and accessible by every backend continues to be an
> issue- it'd be great to improve on that situation too.  I don't think
> having everything encrypted in shared buffers is likely the solution,
> rather, segregating it up might make more sense, again, along similar
> lines to keys and using metadata that's outside of the catalogs, which
> has been discussed previously, though I don't think anyone's actively
> working on it.

What is this trying to protect against?  Without a clear case, I don't
see what that complexity is buying us.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 02:39:44PM -0400, Stephen Frost wrote:
> > > > Of course, we can discuss if what websites do with over-the-wire
> > > > encryption is sensible to compare to what we want to do in PG for
> > > > data-at-rest, but then we shouldn't be talking about what websites do,
> > > > it'd make more sense to look at other data-at-rest encryption systems
> > > > and consider what they're doing.
> > >
> > > (I talked to Joe on chat for clarity.)  In modern TLS, the certificate is
> > > used only for authentication, and Diffie–Hellman is used for key
> > > exchange:
> > >
> > >     https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange
> >
> > Right, and the key that's figured out for each connection is at least
> > specific to the server AND client keys/certificates, thus meaning that
> > they're changed at least as frequently as those change (and clients end
> > up creating ones on the fly randomly if they don't have one, iirc).
> >
> > > So, the question is whether you can pass so much data in TLS that using
> > > the same key for the entire session is a security issue.  TLS originally
> > > had key renegotiation, but that was removed in TLS 1.3:
> > >
> > >     https://www.cloudinsidr.com/content/known-attack-vectors-against-tls-implementation-vulnerabilities/
> > >     To mitigate these types of attacks, TLS 1.3 disallows renegotiation.
> >
> > It was removed due to attacks targeting the renegotiation, not because
> > doing re-keying by itself was a bad idea, or because using multiple keys
> > was seen as a bad idea.
>
> Well, if it was a necessary features, I assume TLS 1.3 would have found
> a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> with a known weakness.

As discussed below- this is about moving goalposts, and that's, in part
at least, why re-keying isn't a *necessary* feature of TLS.  As the
amount of data you transmit over a given TLS connection increases
though, the risk increases and it would be better to re-key.  How much
better?  That depends a great deal on if someone is trying to mount an
attack or not.

> > > Of course, a database is going to process even more data so if the
> > > amount of data encrypted is a problem, we might have a problem too in
> > > using a single key.  This is not related to whether we use one key for
> > > the entire cluster or multiple keys per tablespace --- the problem is
> > > the same.  I guess we could create 1024 keys and use the bottom bits of
> > > the block number to decide what key to use.  However, that still only
> > > pushes the goalposts farther.
> >
> > All of this is about pushing the goalposts farther away, as I see it.
> > There's going to be trade-offs here and there isn't going to be any "one
> > right answer" when it comes to this space.  That's why I'm inclined to
> > argue that we should try to come up with a relatively *good* solution
> > that doesn't create a huge amount of work for us, and then build on
> > that.  To that end, leveraging metadata that we already have outside of
> > the catalogs (databases, tablespaces, potentially other information that
> > we store, essentially, in the filesystem metadata already) to decide on
> > what key to use, and how many we can support, strikes me as a good
> > initial target.
>
> Yes, we will need that for a usable nonce that we don't need to store in
> the blocks and WAL files.

I'm not a fan of the idea of using something which is predictable as a
nonce.  Using the username as the salt for our md5 password mechanism
was, all around, a bad idea.  This seems like it's repeating that
mistake.

> > > Anyway, I will to research the reasonable data size that can be secured
> > > with a single key via AES.  I will look at how PGP encrypts large files
> > > too.
> >
> > This seems unlikely to lead to a definitive result, but it would be
> > interesting to hear if there have been studies around that and what
> > their conclusions were.
>
> I found this:
>
>
https://crypto.stackexchange.com/questions/44113/what-is-a-safe-maximum-message-size-limit-when-encrypting-files-to-disk-with-aes
>     https://crypto.stackexchange.com/questions/20333/encryption-of-big-files-in-java-with-aes-gcm/20340#20340
>
> The numbers listed are:
>
>     Maximum Encrypted Plaintext Size:  68GB
>     Maximum Processed Additional Authenticated Data: 2 x 10^18

These are specific to AES, from a quick reading of those pages, right?

> The 68GB value is "the maximum bits that can be processed with a single
> key/IV(nonce) pair."  We would 8k of data for each 8k page.  If we
> assume a unique nonce per page that is 10^32 bytes.

A unique nonce per page strikes me as excessive...  but then, I think we
should have an actually random nonce rather than something calculated
from the metadata.

> For the WAL we would probably use a different nonce for each 16MB page,
> so we would be OK there too, since that is 10 ^ 36 bytes.
>
> gives us 10^36 bytes before the segment number causes the nonce to
> repeat.

This presumes that the segment number is involved in the nonce
selection, which again strikes me as a bad idea.  Maybe it could be
involved in some way, but we should have a properly random nonce.

> > When it comes to concerns about autovacuum or other system processes,
> > those don't have any direct user connections or interactions, so having
> > them be more privileged and having access to more is reasonable.
>
> Well, I am trying to understand the value of having some keys accessible
> by some parts of the system, and some not.  I am unclear what security
> value that has.

A very real risk is a low-privilege process gaining access to the entire
backend process, and therefore being able to access anything that
backend is able to.

> > Ideally, all of this would leverage a vaulting system or other mechanism
> > which manages access to the keys and allows their usage to be limited.
> > That's been generally accepted as a good way to bridge the gap between
> > having to ask users every time for a key and having keys stored
> > long-term in memory.  Having *only* the keys for the data which the
> > currently connected user is allowed to access would certainly be a great
> > initial capability, even if system processes (including potentially WAL
> > replay) have to have access to all of the keys.  And yes, shared buffers
> > being unencrypted and accessible by every backend continues to be an
> > issue- it'd be great to improve on that situation too.  I don't think
> > having everything encrypted in shared buffers is likely the solution,
> > rather, segregating it up might make more sense, again, along similar
> > lines to keys and using metadata that's outside of the catalogs, which
> > has been discussed previously, though I don't think anyone's actively
> > working on it.
>
> What is this trying to protect against?  Without a clear case, I don't
> see what that complexity is buying us.

This is trying to protect against cross-domain leakage due to specific
security vulnerabilities, similar to those just recently fixed, where a
given backend is able to be comrpomised and is able to be used to run
any code the attacker wishes inside of that backend's process.

If that user-connected backend is only able to access keys/data that is
at the level of their connection, then the data leakage is scoped to
only data at that level.  If they're able to access anything in the
database system, then the entire system and all of the data is
compromised.

Thanks,

Stephen

On Mon, Jul  8, 2019 at 09:30:03PM +0200, Tomas Vondra wrote:
> I think Bruce's proposal was to minimize the time the key is "unlocked"
> in memory by only unlocking them when the user connects and supplies
> some sort of secret (passphrase), and remove them from memory when the
> user disconnects. So there's no way for the auxiliary processes to gain
> access to those keys, because only the user knows the secret.

I mentioned that because I thought that was the security value that
people wanted.  While I can see the value, I don't see how it can be
cleanly accomplished.  Keeping the keys unlocked at all times seems to
be possible, but of much smaller value.

Part of my goal in this discussion is to reverse the rush to implement
and pick apart exactly what is possible, and desirable.

> FWIW I have doubts this scheme actually measurably improves privacy in
> practice, because most busy applications will end up having the keys in
> the memory all the time anyway.

Yep.

> It also assumes memory is unsafe, i.e. bad actors can read it, and
> that's probably a valid concern (root access, vulnerabilities etc.). But
> in that case we already have plenty of issues with data in flight
> anyway, and I doubt TDE is an answer to that.

Agreed.

> > Ideally, all of this would leverage a vaulting system or other mechanism
> > which manages access to the keys and allows their usage to be limited.
> > That's been generally accepted as a good way to bridge the gap between
> > having to ask users every time for a key and having keys stored
> > long-term in memory.
> 
> Right. I agree with this.
> 
> > Having *only* the keys for the data which the
> > currently connected user is allowed to access would certainly be a great
> > initial capability, even if system processes (including potentially WAL
> > replay) have to have access to all of the keys.  And yes, shared buffers
> > being unencrypted and accessible by every backend continues to be an
> > issue- it'd be great to improve on that situation too.  I don't think
> > having everything encrypted in shared buffers is likely the solution,
> > rather, segregating it up might make more sense, again, along similar
> > lines to keys and using metadata that's outside of the catalogs, which
> > has been discussed previously, though I don't think anyone's actively
> > working on it.
> > 
> 
> I very much doubt TDE is a solution to this. Essentially, TDE is a good
> data-at-rest solution, but this seems more like protecting data during
> execution. And in that case I think we may need an entirely different
> encryption scheme.

I thought client-level encryption or pgcrypto-style encryption fits that
need better.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > with a known weakness.
> 
> As discussed below- this is about moving goalposts, and that's, in part
> at least, why re-keying isn't a *necessary* feature of TLS.  As the

I agree we have to allow rekeying and allow multiple unlocked keys in
the server at the same time.  The open question is whether encrypting
different data with different keys and different unlock controls is
possible or useful.

> amount of data you transmit over a given TLS connection increases
> though, the risk increases and it would be better to re-key.  How much
> better?  That depends a great deal on if someone is trying to mount an
> attack or not.

Yep, we need to allow rekey.

> > > > Of course, a database is going to process even more data so if the
> > > > amount of data encrypted is a problem, we might have a problem too in
> > > > using a single key.  This is not related to whether we use one key for
> > > > the entire cluster or multiple keys per tablespace --- the problem is
> > > > the same.  I guess we could create 1024 keys and use the bottom bits of
> > > > the block number to decide what key to use.  However, that still only
> > > > pushes the goalposts farther.
> > > 
> > > All of this is about pushing the goalposts farther away, as I see it.
> > > There's going to be trade-offs here and there isn't going to be any "one
> > > right answer" when it comes to this space.  That's why I'm inclined to
> > > argue that we should try to come up with a relatively *good* solution
> > > that doesn't create a huge amount of work for us, and then build on
> > > that.  To that end, leveraging metadata that we already have outside of
> > > the catalogs (databases, tablespaces, potentially other information that
> > > we store, essentially, in the filesystem metadata already) to decide on
> > > what key to use, and how many we can support, strikes me as a good
> > > initial target.
> > 
> > Yes, we will need that for a usable nonce that we don't need to store in
> > the blocks and WAL files.
> 
> I'm not a fan of the idea of using something which is predictable as a
> nonce.  Using the username as the salt for our md5 password mechanism
> was, all around, a bad idea.  This seems like it's repeating that
> mistake.

Uh, well, renaming the user was a big problem, but that is the only case
I can think of.  I don't see that as an issue for block or WAL sequence
numbers.  If we want to use a different nonce, we have to find a way to
store it or look it up efficiently.  Considering the nonce size, I don't
see how that is possible.

> > > > Anyway, I will to research the reasonable data size that can be secured
> > > > with a single key via AES.  I will look at how PGP encrypts large files
> > > > too.
> > > 
> > > This seems unlikely to lead to a definitive result, but it would be
> > > interesting to hear if there have been studies around that and what
> > > their conclusions were.
> > 
> > I found this:
> > 
> >
https://crypto.stackexchange.com/questions/44113/what-is-a-safe-maximum-message-size-limit-when-encrypting-files-to-disk-with-aes
> >     https://crypto.stackexchange.com/questions/20333/encryption-of-big-files-in-java-with-aes-gcm/20340#20340
> > 
> > The numbers listed are:
> > 
> >     Maximum Encrypted Plaintext Size:  68GB
> >     Maximum Processed Additional Authenticated Data: 2 x 10^18
> 
> These are specific to AES, from a quick reading of those pages, right?

Yes, AES with GCM, which has authentication parts we would not use, so
we would use CBC and CTR, which I think has the same or larger spaces.
> 
> > The 68GB value is "the maximum bits that can be processed with a single
> > key/IV(nonce) pair."  We would 8k of data for each 8k page.  If we
> > assume a unique nonce per page that is 10^32 bytes.
> 
> A unique nonce per page strikes me as excessive...  but then, I think we
> should have an actually random nonce rather than something calculated
> from the metadata.

Uh, well, you are much less likely to get duplicate nonce values by
using block number or WAL sequence number.  If you look at the
implementations, few compute random nonce values.

> > For the WAL we would probably use a different nonce for each 16MB page,
> > so we would be OK there too, since that is 10 ^ 36 bytes.
> > 
> > gives us 10^36 bytes before the segment number causes the nonce to
> > repeat.
> 
> This presumes that the segment number is involved in the nonce
> selection, which again strikes me as a bad idea.  Maybe it could be
> involved in some way, but we should have a properly random nonce.

And you base the random goal on what?  Nonce is number used only once,
and randomness is not a requirement.  You can say you prefer it, but
why, because most implementations don't use random nonce.

> > > When it comes to concerns about autovacuum or other system processes,
> > > those don't have any direct user connections or interactions, so having
> > > them be more privileged and having access to more is reasonable.
> > 
> > Well, I am trying to understand the value of having some keys accessible
> > by some parts of the system, and some not.  I am unclear what security
> > value that has.
> 
> A very real risk is a low-privilege process gaining access to the entire
> backend process, and therefore being able to access anything that
> backend is able to.

Well, if they get to one key, they will get to them all, right?

> > > Ideally, all of this would leverage a vaulting system or other mechanism
> > > which manages access to the keys and allows their usage to be limited.
> > > That's been generally accepted as a good way to bridge the gap between
> > > having to ask users every time for a key and having keys stored
> > > long-term in memory.  Having *only* the keys for the data which the
> > > currently connected user is allowed to access would certainly be a great
> > > initial capability, even if system processes (including potentially WAL
> > > replay) have to have access to all of the keys.  And yes, shared buffers
> > > being unencrypted and accessible by every backend continues to be an
> > > issue- it'd be great to improve on that situation too.  I don't think
> > > having everything encrypted in shared buffers is likely the solution,
> > > rather, segregating it up might make more sense, again, along similar
> > > lines to keys and using metadata that's outside of the catalogs, which
> > > has been discussed previously, though I don't think anyone's actively
> > > working on it.
> > 
> > What is this trying to protect against?  Without a clear case, I don't
> > see what that complexity is buying us.
> 
> This is trying to protect against cross-domain leakage due to specific
> security vulnerabilities, similar to those just recently fixed, where a
> given backend is able to be comrpomised and is able to be used to run
> any code the attacker wishes inside of that backend's process.

I am not sure TLS is a good solution to that.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > > with a known weakness.
> >
> > As discussed below- this is about moving goalposts, and that's, in part
> > at least, why re-keying isn't a *necessary* feature of TLS.  As the
>
> I agree we have to allow rekeying and allow multiple unlocked keys in
> the server at the same time.  The open question is whether encrypting
> different data with different keys and different unlock controls is
> possible or useful.

I'm not sure if there's really a question about if it's *possible*?  As
for if it's useful, I agree there's some debate.

> > amount of data you transmit over a given TLS connection increases
> > though, the risk increases and it would be better to re-key.  How much
> > better?  That depends a great deal on if someone is trying to mount an
> > attack or not.
>
> Yep, we need to allow rekey.

Supporting a way to rekey is definitely a good idea.

> > > > > Of course, a database is going to process even more data so if the
> > > > > amount of data encrypted is a problem, we might have a problem too in
> > > > > using a single key.  This is not related to whether we use one key for
> > > > > the entire cluster or multiple keys per tablespace --- the problem is
> > > > > the same.  I guess we could create 1024 keys and use the bottom bits of
> > > > > the block number to decide what key to use.  However, that still only
> > > > > pushes the goalposts farther.
> > > >
> > > > All of this is about pushing the goalposts farther away, as I see it.
> > > > There's going to be trade-offs here and there isn't going to be any "one
> > > > right answer" when it comes to this space.  That's why I'm inclined to
> > > > argue that we should try to come up with a relatively *good* solution
> > > > that doesn't create a huge amount of work for us, and then build on
> > > > that.  To that end, leveraging metadata that we already have outside of
> > > > the catalogs (databases, tablespaces, potentially other information that
> > > > we store, essentially, in the filesystem metadata already) to decide on
> > > > what key to use, and how many we can support, strikes me as a good
> > > > initial target.
> > >
> > > Yes, we will need that for a usable nonce that we don't need to store in
> > > the blocks and WAL files.
> >
> > I'm not a fan of the idea of using something which is predictable as a
> > nonce.  Using the username as the salt for our md5 password mechanism
> > was, all around, a bad idea.  This seems like it's repeating that
> > mistake.
>
> Uh, well, renaming the user was a big problem, but that is the only case
> I can think of.  I don't see that as an issue for block or WAL sequence
> numbers.  If we want to use a different nonce, we have to find a way to
> store it or look it up efficiently.  Considering the nonce size, I don't
> see how that is possible.

No, this also meant that, as an attacker, I *knew* the salt ahead of
time and therefore could build rainbow tables specifically for that
salt.  I could also use those *same* tables for any system where that
user had an account, even if they used different passwords on different
systems...

I appreciate that *some* of this might not be completely relevant for
the way a nonce is used in cryptography, but I'd be very surprised to
have a cryptographer tell me that a deterministic nonce didn't have
similar issues or didn't reduce the value of the nonce significantly.

> > > > > Anyway, I will to research the reasonable data size that can be secured
> > > > > with a single key via AES.  I will look at how PGP encrypts large files
> > > > > too.
> > > >
> > > > This seems unlikely to lead to a definitive result, but it would be
> > > > interesting to hear if there have been studies around that and what
> > > > their conclusions were.
> > >
> > > I found this:
> > >
> > >
https://crypto.stackexchange.com/questions/44113/what-is-a-safe-maximum-message-size-limit-when-encrypting-files-to-disk-with-aes
> > >     https://crypto.stackexchange.com/questions/20333/encryption-of-big-files-in-java-with-aes-gcm/20340#20340
> > >
> > > The numbers listed are:
> > >
> > >     Maximum Encrypted Plaintext Size:  68GB
> > >     Maximum Processed Additional Authenticated Data: 2 x 10^18
> >
> > These are specific to AES, from a quick reading of those pages, right?
>
> Yes, AES with GCM, which has authentication parts we would not use, so
> we would use CBC and CTR, which I think has the same or larger spaces.
> >
> > > The 68GB value is "the maximum bits that can be processed with a single
> > > key/IV(nonce) pair."  We would 8k of data for each 8k page.  If we
> > > assume a unique nonce per page that is 10^32 bytes.
> >
> > A unique nonce per page strikes me as excessive...  but then, I think we
> > should have an actually random nonce rather than something calculated
> > from the metadata.
>
> Uh, well, you are much less likely to get duplicate nonce values by
> using block number or WAL sequence number.  If you look at the
> implementations, few compute random nonce values.

Which implementations..?  Where do their nonce values come from?  I can
see how a nonce might have to be naturally and deterministically random,
if the source for it is sufficiently varied across the key space, but
starting at '1' and going up with the same key seems like it's just
giving a potential attacker more information about what the encrypted
data contains...

> > > For the WAL we would probably use a different nonce for each 16MB page,
> > > so we would be OK there too, since that is 10 ^ 36 bytes.
> > >
> > > gives us 10^36 bytes before the segment number causes the nonce to
> > > repeat.
> >
> > This presumes that the segment number is involved in the nonce
> > selection, which again strikes me as a bad idea.  Maybe it could be
> > involved in some way, but we should have a properly random nonce.
>
> And you base the random goal on what?  Nonce is number used only once,
> and randomness is not a requirement.  You can say you prefer it, but
> why, because most implementations don't use random nonce.

The encryption schemes I've worked with in the past have used a random
nonce, so I'm wondering where the disconnect is between us on that.

> > > > When it comes to concerns about autovacuum or other system processes,
> > > > those don't have any direct user connections or interactions, so having
> > > > them be more privileged and having access to more is reasonable.
> > >
> > > Well, I am trying to understand the value of having some keys accessible
> > > by some parts of the system, and some not.  I am unclear what security
> > > value that has.
> >
> > A very real risk is a low-privilege process gaining access to the entire
> > backend process, and therefore being able to access anything that
> > backend is able to.
>
> Well, if they get to one key, they will get to them all, right?

That's only the case if all the keys are accessible to a backend process
which is under a user's control.  That would certainly be a bad
situation and one which I'd hope we would avoid.  If the backend that
the user has access to only has access to a subset of the keys, then
while they might be able to access the other encrypted data, they
wouldn't be able to decrypt it.

> > > > Ideally, all of this would leverage a vaulting system or other mechanism
> > > > which manages access to the keys and allows their usage to be limited.
> > > > That's been generally accepted as a good way to bridge the gap between
> > > > having to ask users every time for a key and having keys stored
> > > > long-term in memory.  Having *only* the keys for the data which the
> > > > currently connected user is allowed to access would certainly be a great
> > > > initial capability, even if system processes (including potentially WAL
> > > > replay) have to have access to all of the keys.  And yes, shared buffers
> > > > being unencrypted and accessible by every backend continues to be an
> > > > issue- it'd be great to improve on that situation too.  I don't think
> > > > having everything encrypted in shared buffers is likely the solution,
> > > > rather, segregating it up might make more sense, again, along similar
> > > > lines to keys and using metadata that's outside of the catalogs, which
> > > > has been discussed previously, though I don't think anyone's actively
> > > > working on it.
> > >
> > > What is this trying to protect against?  Without a clear case, I don't
> > > see what that complexity is buying us.
> >
> > This is trying to protect against cross-domain leakage due to specific
> > security vulnerabilities, similar to those just recently fixed, where a
> > given backend is able to be comrpomised and is able to be used to run
> > any code the attacker wishes inside of that backend's process.
>
> I am not sure TLS is a good solution to that.

... TLS?  Or do you mean TDE here..?

I don't mean to throw this up as a requirement on day one of this
feature, rather I'm trying to show where we could potentially go and why
*this* flexibility (supporting a key per tablespace, specifically)
would make sense and how it could help us build a more secure system in
the future.

Thanks,

Stephen

On Mon, Jul  8, 2019 at 06:04:46PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > > > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > > > with a known weakness.
> > > 
> > > As discussed below- this is about moving goalposts, and that's, in part
> > > at least, why re-keying isn't a *necessary* feature of TLS.  As the
> > 
> > I agree we have to allow rekeying and allow multiple unlocked keys in
> > the server at the same time.  The open question is whether encrypting
> > different data with different keys and different unlock controls is
> > possible or useful.
> 
> I'm not sure if there's really a question about if it's *possible*?  As
> for if it's useful, I agree there's some debate.

Right, it is easily possible to keep all keys unlocked, but the value is
minimal, and the complexity will have a cost, which is my point.

> > > amount of data you transmit over a given TLS connection increases
> > > though, the risk increases and it would be better to re-key.  How much
> > > better?  That depends a great deal on if someone is trying to mount an
> > > attack or not.
> > 
> > Yep, we need to allow rekey.
> 
> Supporting a way to rekey is definitely a good idea.

It is a requirement, I think.  We might have problem tracking exactly
what key _version_ each table (or 8k block), or WAL file are.  :-( 
Ideally we would allow only two active keys, and somehow mark each page
as using the odd or even key at a given time, or something strange. 
(Yeah, hand waving here.)

> > Uh, well, renaming the user was a big problem, but that is the only case
> > I can think of.  I don't see that as an issue for block or WAL sequence
> > numbers.  If we want to use a different nonce, we have to find a way to
> > store it or look it up efficiently.  Considering the nonce size, I don't
> > see how that is possible.
> 
> No, this also meant that, as an attacker, I *knew* the salt ahead of
> time and therefore could build rainbow tables specifically for that
> salt.  I could also use those *same* tables for any system where that
> user had an account, even if they used different passwords on different
> systems...

Yes, 'postgres' can be used to create a nice md5 rainbow table that
works on many servers --- good point.  Are rainbow tables possible with
something like AES?

> I appreciate that *some* of this might not be completely relevant for
> the way a nonce is used in cryptography, but I'd be very surprised to
> have a cryptographer tell me that a deterministic nonce didn't have
> similar issues or didn't reduce the value of the nonce significantly.

This post:

    https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm

says:

    GCM is a variation on Counter Mode (CTR).  As you say, with any variant
    of Counter Mode, it is essential  that the Nonce is not repeated with
    the same key.  Hence CTR mode  Nonces often include either a counter or
    a timer element: something that  is guaranteed not to repeat over the
    lifetime of the key.

CTR is what we use for WAL.  8k pages, we would use CBC, which says we
need a random nonce.  I need to dig deeper into ECB mode attack.

> > Uh, well, you are much less likely to get duplicate nonce values by
> > using block number or WAL sequence number.  If you look at the
> > implementations, few compute random nonce values.
> 
> Which implementations..?  Where do their nonce values come from?  I can
> see how a nonce might have to be naturally and deterministically random,
> if the source for it is sufficiently varied across the key space, but
> starting at '1' and going up with the same key seems like it's just
> giving a potential attacker more information about what the encrypted
> data contains...

Well, in many modes the nonce is just a counter, but as stated above,
not all modes.  I need to pull out my security books to remember for
which ones it is safe.  (Frankly, it is a lot easier to use a random
nonce for WAL than 8k pages.)

> > And you base the random goal on what?  Nonce is number used only once,
> > and randomness is not a requirement.  You can say you prefer it, but
> > why, because most implementations don't use random nonce.
> 
> The encryption schemes I've worked with in the past have used a random
> nonce, so I'm wondering where the disconnect is between us on that.

OK.

> > > > > When it comes to concerns about autovacuum or other system processes,
> > > > > those don't have any direct user connections or interactions, so having
> > > > > them be more privileged and having access to more is reasonable.
> > > > 
> > > > Well, I am trying to understand the value of having some keys accessible
> > > > by some parts of the system, and some not.  I am unclear what security
> > > > value that has.
> > > 
> > > A very real risk is a low-privilege process gaining access to the entire
> > > backend process, and therefore being able to access anything that
> > > backend is able to.
> > 
> > Well, if they get to one key, they will get to them all, right?
> 
> That's only the case if all the keys are accessible to a backend process
> which is under a user's control.  That would certainly be a bad
> situation and one which I'd hope we would avoid.  If the backend that
> the user has access to only has access to a subset of the keys, then
> while they might be able to access the other encrypted data, they
> wouldn't be able to decrypt it.

Uh, we already have Postgres security for the data, so what attack
vector has the user reading the RAM, but not seeing all the keys?  Isn't
client-supplied secrets a much better option for this?

> > > > > Ideally, all of this would leverage a vaulting system or other mechanism
> > > > > which manages access to the keys and allows their usage to be limited.
> > > > > That's been generally accepted as a good way to bridge the gap between
> > > > > having to ask users every time for a key and having keys stored
> > > > > long-term in memory.  Having *only* the keys for the data which the
> > > > > currently connected user is allowed to access would certainly be a great
> > > > > initial capability, even if system processes (including potentially WAL
> > > > > replay) have to have access to all of the keys.  And yes, shared buffers
> > > > > being unencrypted and accessible by every backend continues to be an
> > > > > issue- it'd be great to improve on that situation too.  I don't think
> > > > > having everything encrypted in shared buffers is likely the solution,
> > > > > rather, segregating it up might make more sense, again, along similar
> > > > > lines to keys and using metadata that's outside of the catalogs, which
> > > > > has been discussed previously, though I don't think anyone's actively
> > > > > working on it.
> > > > 
> > > > What is this trying to protect against?  Without a clear case, I don't
> > > > see what that complexity is buying us.
> > > 
> > > This is trying to protect against cross-domain leakage due to specific
> > > security vulnerabilities, similar to those just recently fixed, where a
> > > given backend is able to be comrpomised and is able to be used to run
> > > any code the attacker wishes inside of that backend's process.
> > 
> > I am not sure TLS is a good solution to that.
> 
> ... TLS?  Or do you mean TDE here..?

Sorry, yes, TDE.
> 
> I don't mean to throw this up as a requirement on day one of this
> feature, rather I'm trying to show where we could potentially go and why
> *this* flexibility (supporting a key per tablespace, specifically)
> would make sense and how it could help us build a more secure system in
> the future.

Understood.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 7/8/19 6:04 PM, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
>> Uh, well, renaming the user was a big problem, but that is the only case
>> I can think of.  I don't see that as an issue for block or WAL sequence
>> numbers.  If we want to use a different nonce, we have to find a way to
>> store it or look it up efficiently.  Considering the nonce size, I don't
>> see how that is possible.
>
> No, this also meant that, as an attacker, I *knew* the salt ahead of
> time and therefore could build rainbow tables specifically for that
> salt.  I could also use those *same* tables for any system where that
> user had an account, even if they used different passwords on different
> systems...
>
> I appreciate that *some* of this might not be completely relevant for
> the way a nonce is used in cryptography, but I'd be very surprised to
> have a cryptographer tell me that a deterministic nonce didn't have
> similar issues or didn't reduce the value of the nonce significantly.

I have worked side by side on projects with bona fide cryptographers and
I can assure you that they recommended random nonces. Granted, that was
in the early 2000s, but I don't think "modern cryptography" has changed
that any more than "web scale" has made Postgres irrelevant in the
intervening years.

Related links:

https://defuse.ca/cbcmodeiv.htm
https://www.cryptofails.com/post/70059609995/crypto-noobs-1-initialization-vectors


Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 06:04:46PM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> > > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > > > > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > > > > with a known weakness.
> > > >
> > > > As discussed below- this is about moving goalposts, and that's, in part
> > > > at least, why re-keying isn't a *necessary* feature of TLS.  As the
> > >
> > > I agree we have to allow rekeying and allow multiple unlocked keys in
> > > the server at the same time.  The open question is whether encrypting
> > > different data with different keys and different unlock controls is
> > > possible or useful.
> >
> > I'm not sure if there's really a question about if it's *possible*?  As
> > for if it's useful, I agree there's some debate.
>
> Right, it is easily possible to keep all keys unlocked, but the value is
> minimal, and the complexity will have a cost, which is my point.

Having them all unlocked but only accessible to certain privileged
processes if very different from having them unlocked and available to
every backend process.

> > > > amount of data you transmit over a given TLS connection increases
> > > > though, the risk increases and it would be better to re-key.  How much
> > > > better?  That depends a great deal on if someone is trying to mount an
> > > > attack or not.
> > >
> > > Yep, we need to allow rekey.
> >
> > Supporting a way to rekey is definitely a good idea.
>
> It is a requirement, I think.  We might have problem tracking exactly
> what key _version_ each table (or 8k block), or WAL file are.  :-(
> Ideally we would allow only two active keys, and somehow mark each page
> as using the odd or even key at a given time, or something strange.
> (Yeah, hand waving here.)

Well, that wouldn't be the ideal since it would limit us to some small
number of GBs of data written, based on the earlier discussion, right?

I'm not sure that I can see through to a system where we are rewriting
tables that are out on disk every time we hit 60GB of data written.

Or maybe I'm misunderstanding what you're suggesting here..?

> > > Uh, well, renaming the user was a big problem, but that is the only case
> > > I can think of.  I don't see that as an issue for block or WAL sequence
> > > numbers.  If we want to use a different nonce, we have to find a way to
> > > store it or look it up efficiently.  Considering the nonce size, I don't
> > > see how that is possible.
> >
> > No, this also meant that, as an attacker, I *knew* the salt ahead of
> > time and therefore could build rainbow tables specifically for that
> > salt.  I could also use those *same* tables for any system where that
> > user had an account, even if they used different passwords on different
> > systems...
>
> Yes, 'postgres' can be used to create a nice md5 rainbow table that
> works on many servers --- good point.  Are rainbow tables possible with
> something like AES?

I'm not a cryptographer, just to be clear...  but it sure seems like if
you know what the nonce is, and a strong idea about at least what some
of the contents are, then you could work to pre-calculate a portian of
the encrypted data and be able to determine the key based on that.

> > I appreciate that *some* of this might not be completely relevant for
> > the way a nonce is used in cryptography, but I'd be very surprised to
> > have a cryptographer tell me that a deterministic nonce didn't have
> > similar issues or didn't reduce the value of the nonce significantly.
>
> This post:
>
>     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
>
> says:
>
>     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
>     of Counter Mode, it is essential  that the Nonce is not repeated with
>     the same key.  Hence CTR mode  Nonces often include either a counter or
>     a timer element: something that  is guaranteed not to repeat over the
>     lifetime of the key.
>
> CTR is what we use for WAL.  8k pages, we would use CBC, which says we
> need a random nonce.  I need to dig deeper into ECB mode attack.

That page also says:

  Using a random IV / nonce for GCM has been specified as an official
  recommendation by - for instance - NIST. If anybody suggests differently
  then that's up to them.

and a recommendation by NIST certainly holds a lot of water, at least
for me.  They also have a recommendation regarding the amount of data to
encrypt with the same key, and that number is much lower than the 96-bit
randomness of the nonce, with a recommendation to use a
cryptographically sound random, meaning that the chances of a duplicate
are extremely low.

> > > Uh, well, you are much less likely to get duplicate nonce values by
> > > using block number or WAL sequence number.  If you look at the
> > > implementations, few compute random nonce values.
> >
> > Which implementations..?  Where do their nonce values come from?  I can
> > see how a nonce might have to be naturally and deterministically random,
> > if the source for it is sufficiently varied across the key space, but
> > starting at '1' and going up with the same key seems like it's just
> > giving a potential attacker more information about what the encrypted
> > data contains...
>
> Well, in many modes the nonce is just a counter, but as stated above,
> not all modes.  I need to pull out my security books to remember for
> which ones it is safe.  (Frankly, it is a lot easier to use a random
> nonce for WAL than 8k pages.)

I do appreciate that, but given the recommendation that you can encrypt
gigabytes before needing to change, I don't know that we really gain a
lot by changing for every 8K page.

> > > And you base the random goal on what?  Nonce is number used only once,
> > > and randomness is not a requirement.  You can say you prefer it, but
> > > why, because most implementations don't use random nonce.
> >
> > The encryption schemes I've worked with in the past have used a random
> > nonce, so I'm wondering where the disconnect is between us on that.
>
> OK.
>
> > > > > > When it comes to concerns about autovacuum or other system processes,
> > > > > > those don't have any direct user connections or interactions, so having
> > > > > > them be more privileged and having access to more is reasonable.
> > > > >
> > > > > Well, I am trying to understand the value of having some keys accessible
> > > > > by some parts of the system, and some not.  I am unclear what security
> > > > > value that has.
> > > >
> > > > A very real risk is a low-privilege process gaining access to the entire
> > > > backend process, and therefore being able to access anything that
> > > > backend is able to.
> > >
> > > Well, if they get to one key, they will get to them all, right?
> >
> > That's only the case if all the keys are accessible to a backend process
> > which is under a user's control.  That would certainly be a bad
> > situation and one which I'd hope we would avoid.  If the backend that
> > the user has access to only has access to a subset of the keys, then
> > while they might be able to access the other encrypted data, they
> > wouldn't be able to decrypt it.
>
> Uh, we already have Postgres security for the data, so what attack
> vector has the user reading the RAM, but not seeing all the keys?  Isn't
> client-supplied secrets a much better option for this?

I'm all for client-supplied secrets, just to be clear, but much of the
point of this effort is to reduce the burden on the application
developers (after all, that's a lot of what we're doing in the data
layer is for...).

The attack vector, as discussed below, is where the attacker has
complete access to the backend process through some exploit that
bypasses the PG security controls.  We'd like to limit the exposure
from such a situation happening, by having large categories which can't
be breached by even an attacker whose completely compromised a backend.

Note that this will almost certainly involve the kernel, and that's why
multiple shared buffers would be needed, to make it so that a given
backend isn't actually able to access all of shared buffers, but rather,
it's only able to access that portion of the filesystem, and that
portion of shared buffers, and those keys which are able to decrypt the
data that they're, broadly, allowed to see.

Thanks,

Stephen

On Mon, Jul  8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote:
> Yes, 'postgres' can be used to create a nice md5 rainbow table that
> works on many servers --- good point.  Are rainbow tables possible with
> something like AES?
> 
> > I appreciate that *some* of this might not be completely relevant for
> > the way a nonce is used in cryptography, but I'd be very surprised to
> > have a cryptographer tell me that a deterministic nonce didn't have
> > similar issues or didn't reduce the value of the nonce significantly.
> 
> This post:
> 
>     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
> 
> says:
> 
>     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
>     of Counter Mode, it is essential  that the Nonce is not repeated with
>     the same key.  Hence CTR mode  Nonces often include either a counter or
>     a timer element: something that  is guaranteed not to repeat over the
>     lifetime of the key.
> 
> CTR is what we use for WAL.  8k pages, we would use CBC, which says we
> need a random nonce.  I need to dig deeper into ECB mode attack.

Looking here:

    https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm

I think the issues is that we can't use a _counter_ for the nonce since
each page-0 of each table would use the same nonce, and each page-1,
etc.  I assume we would use the table oid and page number as the nonce. 
We can't use the database oid since we copy the files from one database
to another via file system copy and not through the shared buffer cache
where they would be re encrypted.  Using relfilenode seems dangerous. 
For WAL I think it would be the WAL segment number.  It would be nice
to mix that with the "Database system identifier:", but are these the
same on primary and replicas?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul  8, 2019 at 06:43:31PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jul  8, 2019 at 06:04:46PM -0400, Stephen Frost wrote:
> > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> > > > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > > > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > > > > > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > > > > > with a known weakness.
> > > > > 
> > > > > As discussed below- this is about moving goalposts, and that's, in part
> > > > > at least, why re-keying isn't a *necessary* feature of TLS.  As the
> > > > 
> > > > I agree we have to allow rekeying and allow multiple unlocked keys in
> > > > the server at the same time.  The open question is whether encrypting
> > > > different data with different keys and different unlock controls is
> > > > possible or useful.
> > > 
> > > I'm not sure if there's really a question about if it's *possible*?  As
> > > for if it's useful, I agree there's some debate.
> > 
> > Right, it is easily possible to keep all keys unlocked, but the value is
> > minimal, and the complexity will have a cost, which is my point.
> 
> Having them all unlocked but only accessible to certain privileged
> processes if very different from having them unlocked and available to
> every backend process.

Operationally, how would that work?  We unlock them all on boot but
somehow make them inaccessible to some backends after that?

> > > > > amount of data you transmit over a given TLS connection increases
> > > > > though, the risk increases and it would be better to re-key.  How much
> > > > > better?  That depends a great deal on if someone is trying to mount an
> > > > > attack or not.
> > > > 
> > > > Yep, we need to allow rekey.
> > > 
> > > Supporting a way to rekey is definitely a good idea.
> > 
> > It is a requirement, I think.  We might have problem tracking exactly
> > what key _version_ each table (or 8k block), or WAL file are.  :-( 
> > Ideally we would allow only two active keys, and somehow mark each page
> > as using the odd or even key at a given time, or something strange. 
> > (Yeah, hand waving here.)
> 
> Well, that wouldn't be the ideal since it would limit us to some small
> number of GBs of data written, based on the earlier discussion, right?

No, it is GB per secret-nonce combination.

> I'm not sure that I can see through to a system where we are rewriting
> tables that are out on disk every time we hit 60GB of data written.
> 
> Or maybe I'm misunderstanding what you're suggesting here..?

See above.

> > > > Uh, well, renaming the user was a big problem, but that is the only case
> > > > I can think of.  I don't see that as an issue for block or WAL sequence
> > > > numbers.  If we want to use a different nonce, we have to find a way to
> > > > store it or look it up efficiently.  Considering the nonce size, I don't
> > > > see how that is possible.
> > > 
> > > No, this also meant that, as an attacker, I *knew* the salt ahead of
> > > time and therefore could build rainbow tables specifically for that
> > > salt.  I could also use those *same* tables for any system where that
> > > user had an account, even if they used different passwords on different
> > > systems...
> > 
> > Yes, 'postgres' can be used to create a nice md5 rainbow table that
> > works on many servers --- good point.  Are rainbow tables possible with
> > something like AES?
> 
> I'm not a cryptographer, just to be clear...  but it sure seems like if
> you know what the nonce is, and a strong idea about at least what some
> of the contents are, then you could work to pre-calculate a portian of
> the encrypted data and be able to determine the key based on that.

Uh, well, you would think so, but for some reason AES just doesn't allow
that kind of attack, unless you brute force it trying every key.  The
nonce is only to prevent someone from detecting that two output
encryption pages contain the same contents originally.

> > > I appreciate that *some* of this might not be completely relevant for
> > > the way a nonce is used in cryptography, but I'd be very surprised to
> > > have a cryptographer tell me that a deterministic nonce didn't have
> > > similar issues or didn't reduce the value of the nonce significantly.
> > 
> > This post:
> > 
> >     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
> > 
> > says:
> > 
> >     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
> >     of Counter Mode, it is essential  that the Nonce is not repeated with
> >     the same key.  Hence CTR mode  Nonces often include either a counter or
> >     a timer element: something that  is guaranteed not to repeat over the
> >     lifetime of the key.
> > 
> > CTR is what we use for WAL.  8k pages, we would use CBC, which says we
> > need a random nonce.  I need to dig deeper into ECB mode attack.
> 
> That page also says:
> 
>   Using a random IV / nonce for GCM has been specified as an official
>   recommendation by - for instance - NIST. If anybody suggests differently
>   then that's up to them.

Well, if we could generate a random nonce easily, we would do that.  The
question is how important is it for our application.

> and a recommendation by NIST certainly holds a lot of water, at least
> for me.  They also have a recommendation regarding the amount of data to

Agreed.

> > Well, in many modes the nonce is just a counter, but as stated above,
> > not all modes.  I need to pull out my security books to remember for
> > which ones it is safe.  (Frankly, it is a lot easier to use a random
> > nonce for WAL than 8k pages.)
> 
> I do appreciate that, but given the recommendation that you can encrypt
> gigabytes before needing to change, I don't know that we really gain a
> lot by changing for every 8K page.

Uh, well, if you don't do that, you need to use the contents of the
previous page for the next page, and I think we want to encrypt each 8k
page independenty of what was before it.

> > Uh, we already have Postgres security for the data, so what attack
> > vector has the user reading the RAM, but not seeing all the keys?  Isn't
> > client-supplied secrets a much better option for this?
> 
> I'm all for client-supplied secrets, just to be clear, but much of the
> point of this effort is to reduce the burden on the application
> developers (after all, that's a lot of what we're doing in the data
> layer is for...).
> 
> The attack vector, as discussed below, is where the attacker has
> complete access to the backend process through some exploit that
> bypasses the PG security controls.  We'd like to limit the exposure
> from such a situation happening, by having large categories which can't
> be breached by even an attacker whose completely compromised a backend.

As far as I know, TDE was to prevent someone with file system access
from reading the data.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jul  8, 2019 at 06:43:31PM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > On Mon, Jul  8, 2019 at 06:04:46PM -0400, Stephen Frost wrote:
> > > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > > On Mon, Jul  8, 2019 at 05:41:51PM -0400, Stephen Frost wrote:
> > > > > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > > > > Well, if it was a necessary features, I assume TLS 1.3 would have found
> > > > > > > a way to make it secure, no?  Certainly they are not shipping TLS 1.3
> > > > > > > with a known weakness.
> > > > > >
> > > > > > As discussed below- this is about moving goalposts, and that's, in part
> > > > > > at least, why re-keying isn't a *necessary* feature of TLS.  As the
> > > > >
> > > > > I agree we have to allow rekeying and allow multiple unlocked keys in
> > > > > the server at the same time.  The open question is whether encrypting
> > > > > different data with different keys and different unlock controls is
> > > > > possible or useful.
> > > >
> > > > I'm not sure if there's really a question about if it's *possible*?  As
> > > > for if it's useful, I agree there's some debate.
> > >
> > > Right, it is easily possible to keep all keys unlocked, but the value is
> > > minimal, and the complexity will have a cost, which is my point.
> >
> > Having them all unlocked but only accessible to certain privileged
> > processes if very different from having them unlocked and available to
> > every backend process.
>
> Operationally, how would that work?  We unlock them all on boot but
> somehow make them inaccessible to some backends after that?

That could work and doesn't seem like an insurmountable challenge.  The
way that's been discussed, at least somewhere in the past, is leveraging
the exec backend framework to have the user-connected backends work in
an independent space from the processes launched at startup.

> > > > > > amount of data you transmit over a given TLS connection increases
> > > > > > though, the risk increases and it would be better to re-key.  How much
> > > > > > better?  That depends a great deal on if someone is trying to mount an
> > > > > > attack or not.
> > > > >
> > > > > Yep, we need to allow rekey.
> > > >
> > > > Supporting a way to rekey is definitely a good idea.
> > >
> > > It is a requirement, I think.  We might have problem tracking exactly
> > > what key _version_ each table (or 8k block), or WAL file are.  :-(
> > > Ideally we would allow only two active keys, and somehow mark each page
> > > as using the odd or even key at a given time, or something strange.
> > > (Yeah, hand waving here.)
> >
> > Well, that wouldn't be the ideal since it would limit us to some small
> > number of GBs of data written, based on the earlier discussion, right?
>
> No, it is GB per secret-nonce combination.

Hrmpf.  I'm trying to follow the logic that draws this conclusion.

As I understand it, the NIST recommendation is a 96-bit *random* nonce,
and then there's also a recommendation to not encrypt more than 2^32
messages- much less than the 96-bit random nonce, at least potentially
because that limits the repeat-nonce risk to a very low probability.

If the amount-you-can-encrypt is really per secret+nonce combination,
then how do those recommendations make sense..?  This is where I really
think we should be reading through and understanding exactly what the
NIST recommendations are and not just trying to follow through things on
stackoverflow.

> > I'm not sure that I can see through to a system where we are rewriting
> > tables that are out on disk every time we hit 60GB of data written.
> >
> > Or maybe I'm misunderstanding what you're suggesting here..?
>
> See above.

How long would these keys be active for then in the system..?  How much
data would they potentially be used to encrypt?  Strikes me as likely to
be an awful lot...

> > > > > Uh, well, renaming the user was a big problem, but that is the only case
> > > > > I can think of.  I don't see that as an issue for block or WAL sequence
> > > > > numbers.  If we want to use a different nonce, we have to find a way to
> > > > > store it or look it up efficiently.  Considering the nonce size, I don't
> > > > > see how that is possible.
> > > >
> > > > No, this also meant that, as an attacker, I *knew* the salt ahead of
> > > > time and therefore could build rainbow tables specifically for that
> > > > salt.  I could also use those *same* tables for any system where that
> > > > user had an account, even if they used different passwords on different
> > > > systems...
> > >
> > > Yes, 'postgres' can be used to create a nice md5 rainbow table that
> > > works on many servers --- good point.  Are rainbow tables possible with
> > > something like AES?
> >
> > I'm not a cryptographer, just to be clear...  but it sure seems like if
> > you know what the nonce is, and a strong idea about at least what some
> > of the contents are, then you could work to pre-calculate a portian of
> > the encrypted data and be able to determine the key based on that.
>
> Uh, well, you would think so, but for some reason AES just doesn't allow
> that kind of attack, unless you brute force it trying every key.  The
> nonce is only to prevent someone from detecting that two output
> encryption pages contain the same contents originally.

That's certainly interesting, but such a brute-force with every key
would allow it, where, if you use a random nonce, then such an attack
would have to start working only after having access to the data, and
not be something that could be pre-computed.

> > > > I appreciate that *some* of this might not be completely relevant for
> > > > the way a nonce is used in cryptography, but I'd be very surprised to
> > > > have a cryptographer tell me that a deterministic nonce didn't have
> > > > similar issues or didn't reduce the value of the nonce significantly.
> > >
> > > This post:
> > >
> > >     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
> > >
> > > says:
> > >
> > >     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
> > >     of Counter Mode, it is essential  that the Nonce is not repeated with
> > >     the same key.  Hence CTR mode  Nonces often include either a counter or
> > >     a timer element: something that  is guaranteed not to repeat over the
> > >     lifetime of the key.
> > >
> > > CTR is what we use for WAL.  8k pages, we would use CBC, which says we
> > > need a random nonce.  I need to dig deeper into ECB mode attack.
> >
> > That page also says:
> >
> >   Using a random IV / nonce for GCM has been specified as an official
> >   recommendation by - for instance - NIST. If anybody suggests differently
> >   then that's up to them.
>
> Well, if we could generate a random nonce easily, we would do that.  The
> question is how important is it for our application.

[...]

> > and a recommendation by NIST certainly holds a lot of water, at least
> > for me.  They also have a recommendation regarding the amount of data to
>
> Agreed.

This is just it though, at least from my perspective- we are saying "ok,
well, we know people recommend using a random nonce, but that's hard, so
we aren't going to do that because we don't think it's important for our
application", but we aren't cryptographers.  I liken this to whatever
discussion lead to using the username as the salt for our md5
authentication method- great intentions, but not complete understanding,
leading to a less-than-desirable result.

When it comes to this stuff, I don't think we really get to pick and
choose what we follow and what we don't.  If the recommendation from an
authority says we should use random nonces, then we *really* need to
listen and do that, because that authority is a bunch of cryptographers
with a lot more experience and who have definitely spent a great deal
more time thinking about this than we have.

If there's a recommendation from such an authority that says we *don't*
need to use a random nonce, great, I'm happy to go review that and agree
with it, but discussions on stackoverflow or similar don't hold the same
weight that a recommendation from NIST does.

> > > Well, in many modes the nonce is just a counter, but as stated above,
> > > not all modes.  I need to pull out my security books to remember for
> > > which ones it is safe.  (Frankly, it is a lot easier to use a random
> > > nonce for WAL than 8k pages.)
> >
> > I do appreciate that, but given the recommendation that you can encrypt
> > gigabytes before needing to change, I don't know that we really gain a
> > lot by changing for every 8K page.
>
> Uh, well, if you don't do that, you need to use the contents of the
> previous page for the next page, and I think we want to encrypt each 8k
> page independenty of what was before it.

I'm not sure that we really want to do this at the 8K level...  I'll
admit that I'm not completely sure *where* to draw that line then
though.

> > > Uh, we already have Postgres security for the data, so what attack
> > > vector has the user reading the RAM, but not seeing all the keys?  Isn't
> > > client-supplied secrets a much better option for this?
> >
> > I'm all for client-supplied secrets, just to be clear, but much of the
> > point of this effort is to reduce the burden on the application
> > developers (after all, that's a lot of what we're doing in the data
> > layer is for...).
> >
> > The attack vector, as discussed below, is where the attacker has
> > complete access to the backend process through some exploit that
> > bypasses the PG security controls.  We'd like to limit the exposure
> > from such a situation happening, by having large categories which can't
> > be breached by even an attacker whose completely compromised a backend.
>
> As far as I know, TDE was to prevent someone with file system access
> from reading the data.

This seems pretty questionable, doesn't it?  Who gets access to a system
without having some access to what's running at the same time?  Perhaps
if the drive is stolen out from under the running system, but then that
could be protected against using filesystem-level encryption.  If we're
trying to mimic that, which by itself would be good, then wouldn't we
want to do so with similar capabilities- that is, by having
per-tablespace keys?  Since that's what someone running with filesystem
level encryption would have.  Of course, if they don't mount all the
filesystems they've got set up then they have problems, but that's their
choice.

In the end, having this bit of flexibility allows us to have the same
level of options that someone using filesystem-level encryption would
have, but it also starts us down the path to having something which
would work against another attack vector where someone has control over
a complete running backend.

Thanks,

Stephen

On Mon, Jul  8, 2019 at 07:27:12PM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > Operationally, how would that work?  We unlock them all on boot but
> > somehow make them inaccessible to some backends after that?
> 
> That could work and doesn't seem like an insurmountable challenge.  The
> way that's been discussed, at least somewhere in the past, is leveraging
> the exec backend framework to have the user-connected backends work in
> an independent space from the processes launched at startup.

Just do it in another cluster --- why bother with all that?

> > > > > > > amount of data you transmit over a given TLS connection increases
> > > > > > > though, the risk increases and it would be better to re-key.  How much
> > > > > > > better?  That depends a great deal on if someone is trying to mount an
> > > > > > > attack or not.
> > > > > > 
> > > > > > Yep, we need to allow rekey.
> > > > > 
> > > > > Supporting a way to rekey is definitely a good idea.
> > > > 
> > > > It is a requirement, I think.  We might have problem tracking exactly
> > > > what key _version_ each table (or 8k block), or WAL file are.  :-( 
> > > > Ideally we would allow only two active keys, and somehow mark each page
> > > > as using the odd or even key at a given time, or something strange. 
> > > > (Yeah, hand waving here.)
> > > 
> > > Well, that wouldn't be the ideal since it would limit us to some small
> > > number of GBs of data written, based on the earlier discussion, right?
> > 
> > No, it is GB per secret-nonce combination.
> 
> Hrmpf.  I'm trying to follow the logic that draws this conclusion.
> 
> As I understand it, the NIST recommendation is a 96-bit *random* nonce,
> and then there's also a recommendation to not encrypt more than 2^32
> messages- much less than the 96-bit random nonce, at least potentially
> because that limits the repeat-nonce risk to a very low probability.
> 
> If the amount-you-can-encrypt is really per secret+nonce combination,
> then how do those recommendations make sense..?  This is where I really
> think we should be reading through and understanding exactly what the
> NIST recommendations are and not just trying to follow through things on
> stackoverflow.

Yes, it needs more research.

> > > I'm not sure that I can see through to a system where we are rewriting
> > > tables that are out on disk every time we hit 60GB of data written.
> > > 
> > > Or maybe I'm misunderstanding what you're suggesting here..?
> > 
> > See above.
> 
> How long would these keys be active for then in the system..?  How much
> data would they potentially be used to encrypt?  Strikes me as likely to
> be an awful lot...

I think we need to look at CTR vs GCM.

> > Uh, well, you would think so, but for some reason AES just doesn't allow
> > that kind of attack, unless you brute force it trying every key.  The
> > nonce is only to prevent someone from detecting that two output
> > encryption pages contain the same contents originally.
> 
> That's certainly interesting, but such a brute-force with every key
> would allow it, where, if you use a random nonce, then such an attack
> would have to start working only after having access to the data, and
> not be something that could be pre-computed.

Uh, the nonce is going to have to be unecrypted so it can be fed into
the crypto method, so it will be visible.

> > > and a recommendation by NIST certainly holds a lot of water, at least
> > > for me.  They also have a recommendation regarding the amount of data to
> > 
> > Agreed.
> 
> This is just it though, at least from my perspective- we are saying "ok,
> well, we know people recommend using a random nonce, but that's hard, so
> we aren't going to do that because we don't think it's important for our
> application", but we aren't cryptographers.  I liken this to whatever
> discussion lead to using the username as the salt for our md5
> authentication method- great intentions, but not complete understanding,
> leading to a less-than-desirable result.
> 
> When it comes to this stuff, I don't think we really get to pick and
> choose what we follow and what we don't.  If the recommendation from an
> authority says we should use random nonces, then we *really* need to
> listen and do that, because that authority is a bunch of cryptographers
> with a lot more experience and who have definitely spent a great deal
> more time thinking about this than we have.
> 
> If there's a recommendation from such an authority that says we *don't*
> need to use a random nonce, great, I'm happy to go review that and agree
> with it, but discussions on stackoverflow or similar don't hold the same
> weight that a recommendation from NIST does.

Yes, we need to get some experts involved.

> > > > Well, in many modes the nonce is just a counter, but as stated above,
> > > > not all modes.  I need to pull out my security books to remember for
> > > > which ones it is safe.  (Frankly, it is a lot easier to use a random
> > > > nonce for WAL than 8k pages.)
> > > 
> > > I do appreciate that, but given the recommendation that you can encrypt
> > > gigabytes before needing to change, I don't know that we really gain a
> > > lot by changing for every 8K page.
> > 
> > Uh, well, if you don't do that, you need to use the contents of the
> > previous page for the next page, and I think we want to encrypt each 8k
> > page independenty of what was before it.
> 
> I'm not sure that we really want to do this at the 8K level...  I'll
> admit that I'm not completely sure *where* to draw that line then
> though.

Uh, if you want more than 8k you will need to have surrounding 8k pages
in shared buffers, which seems unworkable.

> > As far as I know, TDE was to prevent someone with file system access
> > from reading the data.
> 
> This seems pretty questionable, doesn't it?  Who gets access to a system
> without having some access to what's running at the same time?  Perhaps
> if the drive is stolen out from under the running system, but then that
> could be protected against using filesystem-level encryption.  If we're
> trying to mimic that, which by itself would be good, then wouldn't we
> want to do so with similar capabilities- that is, by having
> per-tablespace keys?  Since that's what someone running with filesystem
> level encryption would have.  Of course, if they don't mount all the
> filesystems they've got set up then they have problems, but that's their
> choice.
> 
> In the end, having this bit of flexibility allows us to have the same
> level of options that someone using filesystem-level encryption would
> have, but it also starts us down the path to having something which
> would work against another attack vector where someone has control over
> a complete running backend.

Again, why not just use a different cluster?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jul 9, 2019 at 3:39 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> BTW how do you know this is what users want? Maybe they do, but then
> again - maybe they just see it as magic and don't realize the extra
> complexity (not just at the database level). In my experience users
> generally want more abstract things, like "Ensure data privacy in case
> media theft," or "protection against evil DBA".
>

I think that it's true that user generally want more abstract things
at system design stage so that's why I've been considering the
functionality of TDE based on security standards such PCI DSS. These
might have a high goal but would be good materials to define
requirements that user will want.

BTW I've created a wiki page[1] for TDE summarizing the discussion. I
will keep it up-to-date but please feel free to update it.

[1] https://wiki.postgresql.org/wiki/Transparent_Data_Encryption

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Jul 08, 2019 at 06:24:40PM -0400, Joe Conway wrote:
>On 7/8/19 6:04 PM, Stephen Frost wrote:
>> * Bruce Momjian (bruce@momjian.us) wrote:
>>> Uh, well, renaming the user was a big problem, but that is the only case
>>> I can think of.  I don't see that as an issue for block or WAL sequence
>>> numbers.  If we want to use a different nonce, we have to find a way to
>>> store it or look it up efficiently.  Considering the nonce size, I don't
>>> see how that is possible.
>>
>> No, this also meant that, as an attacker, I *knew* the salt ahead of
>> time and therefore could build rainbow tables specifically for that
>> salt.  I could also use those *same* tables for any system where that
>> user had an account, even if they used different passwords on different
>> systems...
>>
>> I appreciate that *some* of this might not be completely relevant for
>> the way a nonce is used in cryptography, but I'd be very surprised to
>> have a cryptographer tell me that a deterministic nonce didn't have
>> similar issues or didn't reduce the value of the nonce significantly.
>
>I have worked side by side on projects with bona fide cryptographers and
>I can assure you that they recommended random nonces. Granted, that was
>in the early 2000s, but I don't think "modern cryptography" has changed
>that any more than "web scale" has made Postgres irrelevant in the
>intervening years.
>
>Related links:
>
>https://defuse.ca/cbcmodeiv.htm
>https://www.cryptofails.com/post/70059609995/crypto-noobs-1-initialization-vectors
>

AFAIK it very much depends on the encryption mode. CBC mode does require
random nonces, other modes may be fine with even sequences as long as
the values are not reused. In which case we might even use the LSN, for
example. And I wonder if sha2(LSN) could be considered "random", but
maybe that's entirely silly idea ...


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 08, 2019 at 06:45:50PM -0400, Bruce Momjian wrote:
>On Mon, Jul  8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote:
>> Yes, 'postgres' can be used to create a nice md5 rainbow table that
>> works on many servers --- good point.  Are rainbow tables possible with
>> something like AES?
>>
>> > I appreciate that *some* of this might not be completely relevant for
>> > the way a nonce is used in cryptography, but I'd be very surprised to
>> > have a cryptographer tell me that a deterministic nonce didn't have
>> > similar issues or didn't reduce the value of the nonce significantly.
>>
>> This post:
>>
>>     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
>>
>> says:
>>
>>     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
>>     of Counter Mode, it is essential  that the Nonce is not repeated with
>>     the same key.  Hence CTR mode  Nonces often include either a counter or
>>     a timer element: something that  is guaranteed not to repeat over the
>>     lifetime of the key.
>>
>> CTR is what we use for WAL.  8k pages, we would use CBC, which says we
>> need a random nonce.  I need to dig deeper into ECB mode attack.
>
>Looking here:
>
>    https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
>
>I think the issues is that we can't use a _counter_ for the nonce since
>each page-0 of each table would use the same nonce, and each page-1,
>etc.  I assume we would use the table oid and page number as the nonce.
>We can't use the database oid since we copy the files from one database
>to another via file system copy and not through the shared buffer cache
>where they would be re encrypted.  Using relfilenode seems dangerous.
>For WAL I think it would be the WAL segment number.  It would be nice
>to mix that with the "Database system identifier:", but are these the
>same on primary and replicas?
>

Can't we just store the nonce somewhere? What if for encrypted pages we
only use/encrypt (8kB - X bytes), where X bytes is just enough to store
the nonce and maybe some other encryption metadata (key ID?).

This would be similar to the "special" area on a page, except that that
relies on page header which is encrypted (and thus not accessible before
decrypting the page).

So encryption would:

1) encrypt the (8kB - X bytes) with nonce suitable for the used
   encryption mode (sequence, random, ...)

2) store the nonce / key ID etc. to the reserved space

and encryption would

1) look at the encryption metadata at the end (nonce, key ....)

2) decrypt the page using that info

Or maybe we could define a new relation fork for encrypted relations,
storing all this metadata (not sure if we need that just for the main
fork or for all forks including vm, fsm ...)?


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 08, 2019 at 05:41:55PM -0400, Bruce Momjian wrote:
>On Mon, Jul  8, 2019 at 09:30:03PM +0200, Tomas Vondra wrote:
>> I think Bruce's proposal was to minimize the time the key is "unlocked"
>> in memory by only unlocking them when the user connects and supplies
>> some sort of secret (passphrase), and remove them from memory when the
>> user disconnects. So there's no way for the auxiliary processes to gain
>> access to those keys, because only the user knows the secret.
>
>I mentioned that because I thought that was the security value that
>people wanted.  While I can see the value, I don't see how it can be
>cleanly accomplished.  Keeping the keys unlocked at all times seems to
>be possible, but of much smaller value.
>
>Part of my goal in this discussion is to reverse the rush to implement
>and pick apart exactly what is possible, and desirable.
>
>> FWIW I have doubts this scheme actually measurably improves privacy in
>> practice, because most busy applications will end up having the keys in
>> the memory all the time anyway.
>
>Yep.
>
>> It also assumes memory is unsafe, i.e. bad actors can read it, and
>> that's probably a valid concern (root access, vulnerabilities etc.). But
>> in that case we already have plenty of issues with data in flight
>> anyway, and I doubt TDE is an answer to that.
>
>Agreed.
>
>> > Ideally, all of this would leverage a vaulting system or other mechanism
>> > which manages access to the keys and allows their usage to be limited.
>> > That's been generally accepted as a good way to bridge the gap between
>> > having to ask users every time for a key and having keys stored
>> > long-term in memory.
>>
>> Right. I agree with this.
>>
>> > Having *only* the keys for the data which the
>> > currently connected user is allowed to access would certainly be a great
>> > initial capability, even if system processes (including potentially WAL
>> > replay) have to have access to all of the keys.  And yes, shared buffers
>> > being unencrypted and accessible by every backend continues to be an
>> > issue- it'd be great to improve on that situation too.  I don't think
>> > having everything encrypted in shared buffers is likely the solution,
>> > rather, segregating it up might make more sense, again, along similar
>> > lines to keys and using metadata that's outside of the catalogs, which
>> > has been discussed previously, though I don't think anyone's actively
>> > working on it.
>> >
>>
>> I very much doubt TDE is a solution to this. Essentially, TDE is a good
>> data-at-rest solution, but this seems more like protecting data during
>> execution. And in that case I think we may need an entirely different
>> encryption scheme.
>
>I thought client-level encryption or pgcrypto-style encryption fits that
>need better.
>

I'm not sure client-level encryption is something people really want.

It essentially means moving a lot of the logic to the client. For
example, when you want to do grouping on joins on encrypted columns, we
can't do that in the database when only the client knows the key. And we
know how terribly half-baked those client-side implementations are, even
without the additional encryption complexity.

So it's more a case of not having a better in-database solution :-(

pgcrypto is a bit ugly and tedious, but in general it's a step in the
right direction. The main issue is (a) the key management (or lack of
it) and (2) essentially decrypting once and then processing plaintext in
the rest of the query.

I think (1) could be improved if we had a key vault of some sorts (which
we might get as part of TDE) and (2) might be improved by having special
encrypted data types, with operations offloaded to a trusted execution
environment (TrustZone, SGX, ...).

But that's a very separate topic, unrelated to TDE.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 8, 2019 at 11:20 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Mon, Jul  8, 2019 at 06:04:28PM +0900, Masahiko Sawada wrote:
> > On Sun, Jul 7, 2019 at 1:05 AM Bruce Momjian <bruce@momjian.us> wrote:
> > > What about referential integrity constraints that need to check primary
> > > keys in the encrypted tables?  I also don't see a way of delaying that,
> > > and if you can't do referential integrity into the encrypted tables, it
> > > reduces the value of having encrypted data in the same database rather
> > > than in another database or cluster?
> >
> > I just thought that PostgreSQL's auxiliary processes such as
> > autovacuum, startup, checkpointer, bgwriter should always be able to
> > access all keys because there are already in inside database. Even
> > today these processes don't check any privileges when accessing to
> > data. What security threats we can protect data from by requiring
> > privileges even for auxiliary processes? If this is a security problem
> > isn't it also true for cluster-wide encryption? I guess that processes
> > who have an access privilege on the table can always get the
> > corresponding encryption key. And any processes cannot access an
> > encryption key directly without accessing to a database object.
>
> Well, see my list of three things that users want in an earlier email:
>
>         https://www.postgresql.org/message-id/20190706160514.b67q4f7abcxfdahk@momjian.us
>
> When people are asking for multiple keys (not just for key rotation),
> they are asking to have multiple keys that can be supplied by users only
> when they need to access the data.  Yes, the keys are always in the
> datbase, but the feature request is that they are only unlocked when the
> user needs to access the data.  Obviously, that will not work for
> autovacuum when the encryption is at the block level.

I got your point. I also felt that the client-side encryption or the
encryption during execution (by using pgcrypto with triggers and
views) would fits to these requirements better.

>
> If the key is always unlocked, there is questionable security value of
> having multiple keys, beyond key rotation.
>
> > > I still feel we have not clearly described what the options are:
> > >
> > > 1.  Encrypt everything
> > >
> > > 2.  Encrypt only some tables (for performance reasons), and use only one
> > > key, or use multiple keys to allow for key rotation.  All keys are
> > > always unlocked.
> > >
> > > 3.  Encrypt only some tables with different keys, and the keys are not
> > > always unlocked.
> > >
> > > As Tomas already stated, using tablespaces to distinguish encrypted from
> > > non-encrypted tables doesn't make sense since the file system used for
> > > the storage is immaterial to the encryptions status. An easier way would
> > > be to just add a bit to WAL that would indicate if the rest of the WAL
> > > record is encrypted, though I doubt the performance boost is worth the
> > > complexity.
> >
> > Okay, instead of using tablespaces we can create groups grouping
> > tables being encrypted with the same key. I think the one of the most
> > important point here is to provide a granular encryption feature and
>
> Why is this important?  What are you trying to accomplish?

Because it can not only suppress performance overhead by
encryption/decryption and reduce the amount of data encryption.

The having less number of keys in database would make the
implementation simple, especially for recovery. Since system caches
are not available during recovery we might need a cache mechanism for
keys if we have several thousand keys in database.

>
> > have less the number of keys in database cluster, not to provide per
> > tablespace encryption feature. I'm not going to insist it should be
> > per tablespace encryption.
>
> It is unclear which item you are looking for.  Which number are you
> suggesting from the three listed above in the email URL?
>

Sorry, I'm referring to #2.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 2019-07-08 18:09, Joe Conway wrote:
> In my mind, and in practice to a
> large extent, a postgres tablespace == a unique mount point.

But a critical difference is that in file systems, a separate mount
point has its own journal.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7/9/19 4:34 AM, Tomas Vondra wrote:
> On Mon, Jul 08, 2019 at 06:45:50PM -0400, Bruce Momjian wrote:
>>On Mon, Jul  8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote:
>>> Yes, 'postgres' can be used to create a nice md5 rainbow table that
>>> works on many servers --- good point.  Are rainbow tables possible with
>>> something like AES?
>>>
>>> > I appreciate that *some* of this might not be completely relevant for
>>> > the way a nonce is used in cryptography, but I'd be very surprised to
>>> > have a cryptographer tell me that a deterministic nonce didn't have
>>> > similar issues or didn't reduce the value of the nonce significantly.
>>>
>>> This post:
>>>
>>>     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
>>>
>>> says:
>>>
>>>     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
>>>     of Counter Mode, it is essential  that the Nonce is not repeated with
>>>     the same key.  Hence CTR mode  Nonces often include either a counter or
>>>     a timer element: something that  is guaranteed not to repeat over the
>>>     lifetime of the key.
>>>
>>> CTR is what we use for WAL.  8k pages, we would use CBC, which says we
>>> need a random nonce.  I need to dig deeper into ECB mode attack.
>>
>>Looking here:
>>
>>    https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
>>
>>I think the issues is that we can't use a _counter_ for the nonce since
>>each page-0 of each table would use the same nonce, and each page-1,
>>etc.  I assume we would use the table oid and page number as the nonce.
>>We can't use the database oid since we copy the files from one database
>>to another via file system copy and not through the shared buffer cache
>>where they would be re encrypted.  Using relfilenode seems dangerous.
>>For WAL I think it would be the WAL segment number.  It would be nice
>>to mix that with the "Database system identifier:", but are these the
>>same on primary and replicas?
>>
>
> Can't we just store the nonce somewhere? What if for encrypted pages we
> only use/encrypt (8kB - X bytes), where X bytes is just enough to store
> the nonce and maybe some other encryption metadata (key ID?).
>
> This would be similar to the "special" area on a page, except that that
> relies on page header which is encrypted (and thus not accessible before
> decrypting the page).
>
> So encryption would:
>
> 1) encrypt the (8kB - X bytes) with nonce suitable for the used
>    encryption mode (sequence, random, ...)
>
> 2) store the nonce / key ID etc. to the reserved space
>
> and encryption would
>
> 1) look at the encryption metadata at the end (nonce, key ....)
>
> 2) decrypt the page using that info

That is pretty much what I had been envisioning.

> Or maybe we could define a new relation fork for encrypted relations,
> storing all this metadata (not sure if we need that just for the main
> fork or for all forks including vm, fsm ...)?

I like the idea of a fork if it is workable.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/9/19 4:23 AM, Tomas Vondra wrote:
> On Mon, Jul 08, 2019 at 06:24:40PM -0400, Joe Conway wrote:
>>On 7/8/19 6:04 PM, Stephen Frost wrote:
>>> * Bruce Momjian (bruce@momjian.us) wrote:
>>>> Uh, well, renaming the user was a big problem, but that is the only case
>>>> I can think of.  I don't see that as an issue for block or WAL sequence
>>>> numbers.  If we want to use a different nonce, we have to find a way to
>>>> store it or look it up efficiently.  Considering the nonce size, I don't
>>>> see how that is possible.
>>>
>>> No, this also meant that, as an attacker, I *knew* the salt ahead of
>>> time and therefore could build rainbow tables specifically for that
>>> salt.  I could also use those *same* tables for any system where that
>>> user had an account, even if they used different passwords on different
>>> systems...
>>>
>>> I appreciate that *some* of this might not be completely relevant for
>>> the way a nonce is used in cryptography, but I'd be very surprised to
>>> have a cryptographer tell me that a deterministic nonce didn't have
>>> similar issues or didn't reduce the value of the nonce significantly.
>>
>>I have worked side by side on projects with bona fide cryptographers and
>>I can assure you that they recommended random nonces. Granted, that was
>>in the early 2000s, but I don't think "modern cryptography" has changed
>>that any more than "web scale" has made Postgres irrelevant in the
>>intervening years.
>>
>>Related links:
>>
>>https://defuse.ca/cbcmodeiv.htm
>>https://www.cryptofails.com/post/70059609995/crypto-noobs-1-initialization-vectors
>>
>
> AFAIK it very much depends on the encryption mode. CBC mode does require
> random nonces, other modes may be fine with even sequences as long as
> the values are not reused. In which case we might even use the LSN, for
> example. And I wonder if sha2(LSN) could be considered "random", but
> maybe that's entirely silly idea ...


Yeah, we worked mostly with CBC so that could be the case in terms of
what is required. But I don't think it is ever a bad idea.

But as Stephen pointed out elsewhere on this thread, I think we should
be getting our guidance from places like NIST, which has actual experts
in this stuff.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/9/19 6:07 AM, Peter Eisentraut wrote:
> On 2019-07-08 18:09, Joe Conway wrote:
>> In my mind, and in practice to a
>> large extent, a postgres tablespace == a unique mount point.
>
> But a critical difference is that in file systems, a separate mount
> point has its own journal.

While it would be ideal to have separate WAL, and even separate shared
buffer pools, per tablespace, I think that is too much complexity for
the first implementation and we could have a single separate key for all
WAL for now. The main thing I don't think we want is e.g. a 50TB
database with everything encrypted with a single key -- for the reasons
previously stated.

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/9/19 8:39 AM, Ryan Lambert wrote:
> Hi Thomas,
>
>> CBC mode does require
>> random nonces, other modes may be fine with even sequences as long as
>> the values are not reused.   
>
> I disagree that CBC mode requires random nonces, at least based on what
> NIST has published.  They only require that the IV (not the nonce) must
> be unpredictable per [1]:
>
> " For the CBC and CFB modes, the IVs must be unpredictable."
>
> The unpredictable IV can be generated from a non-random nonce including
> a counter:
>
> "There are two recommended methods for generating unpredictable IVs. The
> first method is to apply the forward cipher function, under the same key
> that is used for the encryption of the plaintext, to a nonce. The nonce
> must be a data block that is unique to each execution of the encryption
> operation. For example, the nonce may be a counter, as described in
> Appendix B, or a message number."
>
> [1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf


The terms nonce and IV are often used more-or-less interchangeably, and
it is important to be clear when we are talking about an IV specifically
- an IV is a specific type of nonce. Nonce means "number used once".
i.e. unique, whereas an IV (for CBC use anyway) should be unique and
random but not necessarily kept secret. The NIST requirements that
Stephen referenced elsewhere on this thread are as I understand it
intended to ensure the random but unique property with high probability.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Tue, Jul  9, 2019 at 08:01:35AM -0400, Joe Conway wrote:
> On 7/9/19 6:07 AM, Peter Eisentraut wrote:
> > On 2019-07-08 18:09, Joe Conway wrote:
> >> In my mind, and in practice to a
> >> large extent, a postgres tablespace == a unique mount point.
> > 
> > But a critical difference is that in file systems, a separate mount
> > point has its own journal.
> 
> While it would be ideal to have separate WAL, and even separate shared
> buffer pools, per tablespace, I think that is too much complexity for
> the first implementation and we could have a single separate key for all
> WAL for now. 

Agreed.  I have thought about this some more.  There is certainly value
in layered security, so if something gets violated, it doesn't open the
whole system.  However, I think the layering has to be done at the right
levels, and I think you want levels that have clear boundaries, like IP
filtering or monitoring.  Placing a boundary inside the database seems
much too complex a level to be effective.  Using separate encrypted and
unencrypted clusters and allowing the encrypted cluster to query the
unencrypted clusters using FDWs does seem like good layering, though the
FDW queries might leak information.

> The main thing I don't think we want is e.g. a 50TB
> database with everything encrypted with a single key -- for the reasons
> previously stated.

Yes, I think we need to research in which cases the nonce must be
random, and how much key space the secret+nonce gives us.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul  8, 2019 at 06:45:50PM -0400, Bruce Momjian wrote:
> On Mon, Jul  8, 2019 at 06:23:13PM -0400, Bruce Momjian wrote:
> > Yes, 'postgres' can be used to create a nice md5 rainbow table that
> > works on many servers --- good point.  Are rainbow tables possible with
> > something like AES?
> > 
> > > I appreciate that *some* of this might not be completely relevant for
> > > the way a nonce is used in cryptography, but I'd be very surprised to
> > > have a cryptographer tell me that a deterministic nonce didn't have
> > > similar issues or didn't reduce the value of the nonce significantly.
> > 
> > This post:
> > 
> >     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
> > 
> > says:
> > 
> >     GCM is a variation on Counter Mode (CTR).  As you say, with any variant
> >     of Counter Mode, it is essential  that the Nonce is not repeated with
> >     the same key.  Hence CTR mode  Nonces often include either a counter or
> >     a timer element: something that  is guaranteed not to repeat over the
> >     lifetime of the key.
> > 
> > CTR is what we use for WAL.  8k pages, we would use CBC, which says we
> > need a random nonce.  I need to dig deeper into ECB mode attack.
> 
> Looking here:
> 
>     https://stackoverflow.com/questions/36760973/why-is-random-iv-fine-for-aes-cbc-but-not-for-aes-gcm
> 
> I think the issues is that we can't use a _counter_ for the nonce since
> each page-0 of each table would use the same nonce, and each page-1,
> etc.  I assume we would use the table oid and page number as the nonce. 
> We can't use the database oid since we copy the files from one database
> to another via file system copy and not through the shared buffer cache
> where they would be re encrypted.  Using relfilenode seems dangerous. 

FYI, pg_upgrade already preserves the pg_class.oid, which is why I
recommended it over pg_class.relfilenode:


https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/bin/pg_upgrade/pg_upgrade.c;h=ff78929707ef12699a7579274693f6020c54c755;hb=HEAD#l14

    We control all assignments of pg_class.oid (and relfilenode) so toast
    oids are the same between old and new clusters.  This is important
    because toast oids are stored as toast pointers in user tables.
    
    While pg_class.oid and pg_class.relfilenode are initially the same
    in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM
    FULL.  In the new cluster, pg_class.oid and pg_class.relfilenode will
    be the same and will match the old pg_class.oid value.  Because of
    this, old/new pg_class.relfilenode values will not match if CLUSTER,
    REINDEX, or VACUUM FULL have been performed in the old cluster.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul  8, 2019 at 09:57:57PM -0600, Ryan Lambert wrote:
> Hey everyone,
> 
> Here is my input regarding nonces and randomness.
> 
> > As I understand it, the NIST recommendation is a 96-bit *random* nonce,
> 
> I could not find that exact requirement in the NIST documents, though given the
> volume of that library it would be possible to miss.  The recommendation I
> repeatedly saw for the nonce was unique.  There is also an important
> distinction that the nonce is not the Initialization Vector (IV), it can be
> used as part of the IV, more on that later.
> 
> The most succinct definition for nonce I found was in SP-800-38A [1] page 4:
>  "A value that is only used once."
> SP-800-90A [2] (page 6) expands on the definition: "A time-varying value that
> has at most a negligible chance of repeating, e.g., a random value that is
> generated anew for each use, a timestamp, a sequence number, or some
> combination of these."
> 
> The second definition references randomness but does not require it.  [1] (pg
> 19) reinforces the importance of uniqueness:  "A procedure should be
> established to ensure the uniqueness of the message nonces"

Yes, that is what I remembered but the URL I referenced stated
randomness is preferred.  I was hopeful that whatever was preferring
randomness was trying to avoid a problem we didn't have.

> Further, knowing the nonce gets you nowhere, it isn't the salt until it is ran
> through the forward cipher with the encryption key.  Even with the nonce the
> attacker has doesn't know the salt unless they steal the key, and that's a
> bigger problem.

Yes, I had forgotten about that step --- good point, meaning that the
nonce for block zero is different for every encryption key.

> The strictest definition of nonce I found was in [2] (pg 19) defining nonces to
> use in the process of random generation:
> 
> "The nonce shall be either:
> a. A value with at least (security_strength/2) bits of entropy, or
> b. A value that is expected to repeat no more often than a (security_strength/
> 2)-bit random
> string would be expected to repeat."
> 
> Even there it is randomness (a) or uniqueness (b).

Thanks, this was very helpful.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jul  9, 2019 at 10:34:06AM +0200, Tomas Vondra wrote:
> > I think the issues is that we can't use a _counter_ for the nonce since
> > each page-0 of each table would use the same nonce, and each page-1,
> > etc.  I assume we would use the table oid and page number as the nonce.
> > We can't use the database oid since we copy the files from one database
> > to another via file system copy and not through the shared buffer cache
> > where they would be re encrypted.  Using relfilenode seems dangerous.
> > For WAL I think it would be the WAL segment number.  It would be nice
> > to mix that with the "Database system identifier:", but are these the
> > same on primary and replicas?
> > 
> 
> Can't we just store the nonce somewhere? What if for encrypted pages we
> only use/encrypt (8kB - X bytes), where X bytes is just enough to store
> the nonce and maybe some other encryption metadata (key ID?).

Storing the nonce on each 8k page is going to add complexity, so I am
trying to figure out if it is a security requirement.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jul  9, 2019 at 08:01:35AM -0400, Joe Conway wrote:
> > On 7/9/19 6:07 AM, Peter Eisentraut wrote:
> > > On 2019-07-08 18:09, Joe Conway wrote:
> > >> In my mind, and in practice to a
> > >> large extent, a postgres tablespace == a unique mount point.
> > >
> > > But a critical difference is that in file systems, a separate mount
> > > point has its own journal.
> >
> > While it would be ideal to have separate WAL, and even separate shared
> > buffer pools, per tablespace, I think that is too much complexity for
> > the first implementation and we could have a single separate key for all
> > WAL for now.

I agree that all of that isn't necessary for an initial implementation,
I was rather trying to lay out how we could improve on this in the
future and why having the keying done at a tablespace level makes sense
initially because we can then potentially move forward with further
segregation to improve the situation.  I do believe it's also useful in
its own right, to be clear, just not as nice since a compromised backend
could still get access to data in shared buffers that it really
shouldn't be able to, even broadly, see.

> Agreed.  I have thought about this some more.  There is certainly value
> in layered security, so if something gets violated, it doesn't open the
> whole system.  However, I think the layering has to be done at the right
> levels, and I think you want levels that have clear boundaries, like IP
> filtering or monitoring.  Placing a boundary inside the database seems
> much too complex a level to be effective.  Using separate encrypted and
> unencrypted clusters and allowing the encrypted cluster to query the
> unencrypted clusters using FDWs does seem like good layering, though the
> FDW queries might leak information.

Using FDWs simply isn't a solution to this, for a few different reasons-
the first is that our solution to authentication for FDWs is to store
passwords in our catalog tables, but an FDW table also doesn't behave
like a regular table in many important cases.

> > The main thing I don't think we want is e.g. a 50TB
> > database with everything encrypted with a single key -- for the reasons
> > previously stated.
>
> Yes, I think we need to research in which cases the nonce must be
> random, and how much key space the secret+nonce gives us.

Agreed on both.

Thanks,

Stephen

On Tue, Jul  9, 2019 at 10:59:12AM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> I agree that all of that isn't necessary for an initial implementation,
> I was rather trying to lay out how we could improve on this in the
> future and why having the keying done at a tablespace level makes sense
> initially because we can then potentially move forward with further
> segregation to improve the situation.  I do believe it's also useful in
> its own right, to be clear, just not as nice since a compromised backend
> could still get access to data in shared buffers that it really
> shouldn't be able to, even broadly, see.

I think TDE is feature of questionable value at best and the idea that
we would fundmentally change the internals of Postgres to add more
features to it seems very unlikely.  I realize we have to discuss it so
we don't block reasonable future feature development.

> > Agreed.  I have thought about this some more.  There is certainly value
> > in layered security, so if something gets violated, it doesn't open the
> > whole system.  However, I think the layering has to be done at the right
> > levels, and I think you want levels that have clear boundaries, like IP
> > filtering or monitoring.  Placing a boundary inside the database seems
> > much too complex a level to be effective.  Using separate encrypted and
> > unencrypted clusters and allowing the encrypted cluster to query the
> > unencrypted clusters using FDWs does seem like good layering, though the
> > FDW queries might leak information.
> 
> Using FDWs simply isn't a solution to this, for a few different reasons-
> the first is that our solution to authentication for FDWs is to store
> passwords in our catalog tables, but an FDW table also doesn't behave
> like a regular table in many important cases.

The FDW authentication problem is something I think we need to improve
no matter what.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jul  9, 2019 at 09:16:17AM -0400, Joe Conway wrote:
> On 7/9/19 8:39 AM, Ryan Lambert wrote:
> > Hi Thomas,
> > 
> >> CBC mode does require
> >> random nonces, other modes may be fine with even sequences as long as
> >> the values are not reused.   
> > 
> > I disagree that CBC mode requires random nonces, at least based on what
> > NIST has published.  They only require that the IV (not the nonce) must
> > be unpredictable per [1]:
> > 
> > " For the CBC and CFB modes, the IVs must be unpredictable."
> > 
> > The unpredictable IV can be generated from a non-random nonce including
> > a counter:
> > 
> > "There are two recommended methods for generating unpredictable IVs. The
> > first method is to apply the forward cipher function, under the same key
> > that is used for the encryption of the plaintext, to a nonce. The nonce
> > must be a data block that is unique to each execution of the encryption
> > operation. For example, the nonce may be a counter, as described in
> > Appendix B, or a message number."
> > 
> > [1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> 
> 
> The terms nonce and IV are often used more-or-less interchangeably, and
> it is important to be clear when we are talking about an IV specifically
> - an IV is a specific type of nonce. Nonce means "number used once".
> i.e. unique, whereas an IV (for CBC use anyway) should be unique and
> random but not necessarily kept secret. The NIST requirements that
> Stephen referenced elsewhere on this thread are as I understand it
> intended to ensure the random but unique property with high probability.

Good point about nonce and IV.  I wonder if running the nonce through
the cipher with the key makes it random enough to use as an IV.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jul  9, 2019 at 10:59:12AM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > I agree that all of that isn't necessary for an initial implementation,
> > I was rather trying to lay out how we could improve on this in the
> > future and why having the keying done at a tablespace level makes sense
> > initially because we can then potentially move forward with further
> > segregation to improve the situation.  I do believe it's also useful in
> > its own right, to be clear, just not as nice since a compromised backend
> > could still get access to data in shared buffers that it really
> > shouldn't be able to, even broadly, see.
>
> I think TDE is feature of questionable value at best and the idea that
> we would fundmentally change the internals of Postgres to add more
> features to it seems very unlikely.  I realize we have to discuss it so
> we don't block reasonable future feature development.

We'd be getting to something much better than just TDE by going down
that road- we'd be able to properly leverage the kernel to enforce real
MAC.  I get that this would be a change but I'm not entirely convinced
that it'd be as much of a fundamental change as implied here.  I expect
that we're going to get to a point where we want to have multiple shared
buffer segments for other reasons anyway.

> > > Agreed.  I have thought about this some more.  There is certainly value
> > > in layered security, so if something gets violated, it doesn't open the
> > > whole system.  However, I think the layering has to be done at the right
> > > levels, and I think you want levels that have clear boundaries, like IP
> > > filtering or monitoring.  Placing a boundary inside the database seems
> > > much too complex a level to be effective.  Using separate encrypted and
> > > unencrypted clusters and allowing the encrypted cluster to query the
> > > unencrypted clusters using FDWs does seem like good layering, though the
> > > FDW queries might leak information.
> >
> > Using FDWs simply isn't a solution to this, for a few different reasons-
> > the first is that our solution to authentication for FDWs is to store
> > passwords in our catalog tables, but an FDW table also doesn't behave
> > like a regular table in many important cases.
>
> The FDW authentication problem is something I think we need to improve
> no matter what.

Yes, constrained delegation with Kerberos would certainly be an
improvement, and having a way to do something like peer auth when local,
and maybe even a server-to-server trust based on certificates or similar
might be an option.

Thanks,

Stephen

On 7/9/19 11:11 AM, Bruce Momjian wrote:
> On Tue, Jul  9, 2019 at 09:16:17AM -0400, Joe Conway wrote:
>> On 7/9/19 8:39 AM, Ryan Lambert wrote:
>> > Hi Thomas,
>> >
>> >> CBC mode does require
>> >> random nonces, other modes may be fine with even sequences as long as
>> >> the values are not reused.   
>> >
>> > I disagree that CBC mode requires random nonces, at least based on what
>> > NIST has published.  They only require that the IV (not the nonce) must
>> > be unpredictable per [1]:
>> >
>> > " For the CBC and CFB modes, the IVs must be unpredictable."
>> >
>> > The unpredictable IV can be generated from a non-random nonce including
>> > a counter:
>> >
>> > "There are two recommended methods for generating unpredictable IVs. The
>> > first method is to apply the forward cipher function, under the same key
>> > that is used for the encryption of the plaintext, to a nonce. The nonce
>> > must be a data block that is unique to each execution of the encryption
>> > operation. For example, the nonce may be a counter, as described in
>> > Appendix B, or a message number."
>> >
>> > [1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
>>
>>
>> The terms nonce and IV are often used more-or-less interchangeably, and
>> it is important to be clear when we are talking about an IV specifically
>> - an IV is a specific type of nonce. Nonce means "number used once".
>> i.e. unique, whereas an IV (for CBC use anyway) should be unique and
>> random but not necessarily kept secret. The NIST requirements that
>> Stephen referenced elsewhere on this thread are as I understand it
>> intended to ensure the random but unique property with high probability.
>
> Good point about nonce and IV.  I wonder if running the nonce through
> the cipher with the key makes it random enough to use as an IV.

Based on that NIST document it seems so.

The trick will be to be 100% sure we never reuse a nonce that is used to
produce the IV when using the same key.

I think the potential to get that wrong (i.e. inadvertently reuse a
nonce) would lead to using the second described method

  "The second method is to generate a random data block using a
   FIPS-approved random number generator."

That method is what I am used to seeing. But with the second method we
need to store the IV, with the first we could reproduce it if we select
our initial nonce carefully.

So thinking out loud, and perhaps you already said this Bruce, but I
guess the input nonce used to generate the IV could be something like
pg_class.oid and blocknum concatenated together with some delimiting
character as long as we guarantee that we generate different keys in
different databases. Then there would be no need to store the IV since
we could reproduce it. This all assumes that we encrypt each block
independently. Sound correct?

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:
> On 7/9/19 11:11 AM, Bruce Momjian wrote:
> > Good point about nonce and IV.  I wonder if running the nonce
> > through the cipher with the key makes it random enough to use as an
> > IV.
>
> Based on that NIST document it seems so.
>
> The trick will be to be 100% sure we never reuse a nonce that is used
> to produce the IV when using the same key.
>
> I think the potential to get that wrong (i.e. inadvertently reuse a
> nonce) would lead to using the second described method
>
>   "The second method is to generate a random data block using a
>   FIPS-approved random number generator."
>
> That method is what I am used to seeing. But with the second method
> we need to store the IV, with the first we could reproduce it if we
> select our initial nonce carefully.
>
> So thinking out loud, and perhaps you already said this Bruce, but I
> guess the input nonce used to generate the IV could be something like
> pg_class.oid and blocknum concatenated together with some delimiting
> character as long as we guarantee that we generate different keys in
> different databases. Then there would be no need to store the IV since
> we could reproduce it.

Uh, yes, and no.  Yes, we can use the pg_class.oid (since it has to
be preserved by pg_upgrade anyway), and the page number.  However,
different databases can have the same pg_class.oid/page number
combination, so there would be duplication between databases.  Now, you
might say let's add the pg_database.oid, but unfortunately, because of
the way we file-system-copy files from one database to another during
database creation (it doesn't go through shared buffers), we can't use
pg_database.oid as part of the nonce.

My only idea here is that we actually decrypt/re-encrypted pages as we
copy them at the file system level during database creation to match the
new pg_database.oid.  This would allow pg_database.oid in the nonce/IV.
(I think we will need to modify pg_upgrade to preserve pg_database.oid.)

If the nonce/IV is 96 bits, then that is 12 bytes or 3 4-byte values.
pg_class.oid is 4 bytes, pg_database.oid is 4 bytes, and that leaves
4-bytes for the block number, which gets us to 32TB before the page
counter would overflow a 4-byte value, and our max table size is 32TB
anyway, so that all works.

> This all assumes that we encrypt each block independently. Sound
> correct?

Yes, I think 8k encryption granularity is a requirement.  If not, you
would need to potentially load and write multiple 8k pages for a single
8k page change, which seems very complex.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 7/9/19 3:50 PM, Bruce Momjian wrote:
> On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:
>> On 7/9/19 11:11 AM, Bruce Momjian wrote:
>> > Good point about nonce and IV.  I wonder if running the nonce
>> > through the cipher with the key makes it random enough to use as an
>> > IV.
>>
>> Based on that NIST document it seems so.
>>
>> The trick will be to be 100% sure we never reuse a nonce that is used
>> to produce the IV when using the same key.
>>
>> I think the potential to get that wrong (i.e. inadvertently reuse a
>> nonce) would lead to using the second described method
>>
>>   "The second method is to generate a random data block using a
>>   FIPS-approved random number generator."
>>
>> That method is what I am used to seeing. But with the second method
>> we need to store the IV, with the first we could reproduce it if we
>> select our initial nonce carefully.
>>
>> So thinking out loud, and perhaps you already said this Bruce, but I
>> guess the input nonce used to generate the IV could be something like
>> pg_class.oid and blocknum concatenated together with some delimiting
>> character as long as we guarantee that we generate different keys in
>> different databases. Then there would be no need to store the IV since
>> we could reproduce it.
>
> Uh, yes, and no.  Yes, we can use the pg_class.oid (since it has to
> be preserved by pg_upgrade anyway), and the page number.  However,
> different databases can have the same pg_class.oid/page number
> combination, so there would be duplication between databases.

But as I said "as long as we guarantee that we generate different keys
in different databases". The IV only needs to be unique for a given key.
the combination of oid and page number when run through the cipher
should always produce a unique IV with high probability. And if we
generate random keys with sufficient entropy the chances of collision
should approach zero.

> If the nonce/IV is 96 bits, then that is 12 bytes or 3 4-byte values.
> pg_class.oid is 4 bytes, pg_database.oid is 4 bytes, and that leaves
> 4-bytes for the block number, which gets us to 32TB before the page
> counter would overflow a 4-byte value, and our max table size is 32TB
> anyway, so that all works.

The IV will be the same as the algorithm block size (128 bits for AES).
It gets XOR'd with the first block of plaintext in CBC. The nonce used
to produce the IV does not need to be the same size, but but running it
through the cipher the output IV will be the correct size.

https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
https://en.wikipedia.org/wiki/Initialization_vector

>> This all assumes that we encrypt each block independently. Sound
>> correct?
>
> Yes, I think 8k encryption granularity is a requirement.  If not, you
> would need to potentially load and write multiple 8k pages for a single
> 8k page change, which seems very complex.

Exactly, and it would also be terrible for performance.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote:
>On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:
>> On 7/9/19 11:11 AM, Bruce Momjian wrote:
>> > Good point about nonce and IV.  I wonder if running the nonce
>> > through the cipher with the key makes it random enough to use as an
>> > IV.
>>
>> Based on that NIST document it seems so.
>>
>> The trick will be to be 100% sure we never reuse a nonce that is used
>> to produce the IV when using the same key.
>>
>> I think the potential to get that wrong (i.e. inadvertently reuse a
>> nonce) would lead to using the second described method
>>
>>   "The second method is to generate a random data block using a
>>   FIPS-approved random number generator."
>>
>> That method is what I am used to seeing. But with the second method
>> we need to store the IV, with the first we could reproduce it if we
>> select our initial nonce carefully.
>>
>> So thinking out loud, and perhaps you already said this Bruce, but I
>> guess the input nonce used to generate the IV could be something like
>> pg_class.oid and blocknum concatenated together with some delimiting
>> character as long as we guarantee that we generate different keys in
>> different databases. Then there would be no need to store the IV since
>> we could reproduce it.
>
>Uh, yes, and no.  Yes, we can use the pg_class.oid (since it has to
>be preserved by pg_upgrade anyway), and the page number.  However,
>different databases can have the same pg_class.oid/page number
>combination, so there would be duplication between databases.  Now, you
>might say let's add the pg_database.oid, but unfortunately, because of
>the way we file-system-copy files from one database to another during
>database creation (it doesn't go through shared buffers), we can't use
>pg_database.oid as part of the nonce.
>
>My only idea here is that we actually decrypt/re-encrypted pages as we
>copy them at the file system level during database creation to match the
>new pg_database.oid.  This would allow pg_database.oid in the nonce/IV.
>(I think we will need to modify pg_upgrade to preserve pg_database.oid.)
>

Ot you could just encrypt them with a different key, and you would not
need to make database OID part of the nonce.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 7/9/19 4:12 PM, Tomas Vondra wrote:
> On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote:
>>On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:

>>> the input nonce used to generate the IV could be something like
>>> pg_class.oid and blocknum concatenated together with some delimiting
>>> character as long as we guarantee that we generate different keys in
>>> different databases.

<snip>

> Ot you could just encrypt them with a different key, and you would not
> need to make database OID part of the nonce.

Yeah that was pretty much exactly what I was trying to say above ;-)

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 2019-Jul-09, Joe Conway wrote:

> > Ot you could just encrypt them with a different key, and you would not
> > need to make database OID part of the nonce.
> 
> Yeah that was pretty much exactly what I was trying to say above ;-)

So you need to decrypt each file and encrypt again when doing CREATE
DATABASE?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote:
>On 2019-Jul-09, Joe Conway wrote:
>
>> > Ot you could just encrypt them with a different key, and you would not
>> > need to make database OID part of the nonce.
>>
>> Yeah that was pretty much exactly what I was trying to say above ;-)
>
>So you need to decrypt each file and encrypt again when doing CREATE
>DATABASE?
>

The question is whether we actually need to do that? Do we change OIDs
of relations when creating the database? If not, we don't need to
re-encrypt because having copies of the same block encrypted with the
same nonce is not an issue (just like copying encrypted files is not an
issue).

Of course, we may need a CREATE DATABASE option that would force
re-encryption with a different key, but it's not necessary because of
nonces or whatnot.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 2019-Jul-09, Tomas Vondra wrote:

> On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote:
> > On 2019-Jul-09, Joe Conway wrote:
> > 
> > > > Ot you could just encrypt them with a different key, and you would not
> > > > need to make database OID part of the nonce.
> > > 
> > > Yeah that was pretty much exactly what I was trying to say above ;-)
> > 
> > So you need to decrypt each file and encrypt again when doing CREATE
> > DATABASE?
> 
> The question is whether we actually need to do that?

I mean if the new database is supposed to be encrypted with key B, you
can't just copy the files from the other database, since they are
encrypted with key A, right?  Even if you consider that both copies of
each table have the same OID and each block has the same nonce.

> Do we change OIDs of relations when creating the database? If not, we
> don't need to re-encrypt because having copies of the same block
> encrypted with the same nonce is not an issue (just like copying
> encrypted files is not an issue).

Are you thinking that the files can be decrypted by the two keys
somehow?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote:
> >On 2019-Jul-09, Joe Conway wrote:
> >
> >>> Ot you could just encrypt them with a different key, and you would not
> >>> need to make database OID part of the nonce.
> >>
> >>Yeah that was pretty much exactly what I was trying to say above ;-)
> >
> >So you need to decrypt each file and encrypt again when doing CREATE
> >DATABASE?
>
> The question is whether we actually need to do that? Do we change OIDs
> of relations when creating the database? If not, we don't need to
> re-encrypt because having copies of the same block encrypted with the
> same nonce is not an issue (just like copying encrypted files is not an
> issue).
>
> Of course, we may need a CREATE DATABASE option that would force
> re-encryption with a different key, but it's not necessary because of
> nonces or whatnot.

This also depends on if we actually encrypt the template databases.
Seems like that could be optional, if we're supporting different keys
for different databases.

In that case we'd need the "encrypt this database" option during CREATE
DATABASE, of course.

Thanks,

Stephen

On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote:
>On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:
>> On 7/9/19 11:11 AM, Bruce Momjian wrote:
>> > Good point about nonce and IV.  I wonder if running the nonce
>> > through the cipher with the key makes it random enough to use as an
>> > IV.
>>
>> Based on that NIST document it seems so.
>>
>> The trick will be to be 100% sure we never reuse a nonce that is used
>> to produce the IV when using the same key.
>>
>> I think the potential to get that wrong (i.e. inadvertently reuse a
>> nonce) would lead to using the second described method
>>
>>   "The second method is to generate a random data block using a
>>   FIPS-approved random number generator."
>>
>> That method is what I am used to seeing. But with the second method
>> we need to store the IV, with the first we could reproduce it if we
>> select our initial nonce carefully.
>>
>> So thinking out loud, and perhaps you already said this Bruce, but I
>> guess the input nonce used to generate the IV could be something like
>> pg_class.oid and blocknum concatenated together with some delimiting
>> character as long as we guarantee that we generate different keys in
>> different databases. Then there would be no need to store the IV since
>> we could reproduce it.
>
>Uh, yes, and no.  Yes, we can use the pg_class.oid (since it has to
>be preserved by pg_upgrade anyway), and the page number.  However,
>different databases can have the same pg_class.oid/page number
>combination, so there would be duplication between databases.  Now, you
>might say let's add the pg_database.oid, but unfortunately, because of
>the way we file-system-copy files from one database to another during
>database creation (it doesn't go through shared buffers), we can't use
>pg_database.oid as part of the nonce.
>
>My only idea here is that we actually decrypt/re-encrypted pages as we
>copy them at the file system level during database creation to match the
>new pg_database.oid.  This would allow pg_database.oid in the nonce/IV.
>(I think we will need to modify pg_upgrade to preserve pg_database.oid.)
>
>If the nonce/IV is 96 bits, then that is 12 bytes or 3 4-byte values.
>pg_class.oid is 4 bytes, pg_database.oid is 4 bytes, and that leaves
>4-bytes for the block number, which gets us to 32TB before the page
>counter would overflow a 4-byte value, and our max table size is 32TB
>anyway, so that all works.
>

I don't think that works, because that'd mean we're encrypting the same
page with the same nonce over and over, which means reusing the reuse
(even if you hash/encrypt it). Or did I miss something?

There are two basic ways to construct nonces - CSPRNG and sequences, and
then a combination of both, i.e. one part is generated from a sequence
and one randomly.

FWIW not sure using OIDs as nonces directly is a good idea, as those are
inherently low entropy data - how often do you see databases with OIDs
above 1M or so? Probably not very often, and in most cases those are
databases where those OIDs are for OIDs and large objects, so irrelevant
for this purpose. I might be wrong but having a 96-bit nonce with maybe
just 32bits of entrophy seems suspicious.

That does not mean we can't use the OIDs at all, but maybe hashing them
into a single 4B value, and then picking the remaining 8B randomly.
Also, we have a "natural" sequence in the database - LSNs, maybe that
would be a good source of nonces too?


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Tue, Jul 09, 2019 at 05:31:49PM -0400, Alvaro Herrera wrote:
>On 2019-Jul-09, Tomas Vondra wrote:
>
>> On Tue, Jul 09, 2019 at 05:06:45PM -0400, Alvaro Herrera wrote:
>> > On 2019-Jul-09, Joe Conway wrote:
>> >
>> > > > Ot you could just encrypt them with a different key, and you would not
>> > > > need to make database OID part of the nonce.
>> > >
>> > > Yeah that was pretty much exactly what I was trying to say above ;-)
>> >
>> > So you need to decrypt each file and encrypt again when doing CREATE
>> > DATABASE?
>>
>> The question is whether we actually need to do that?
>
>I mean if the new database is supposed to be encrypted with key B, you
>can't just copy the files from the other database, since they are
>encrypted with key A, right?  Even if you consider that both copies of
>each table have the same OID and each block has the same nonce.
>

Sure, if the databases are supposed to be encrypted with different keys,
then we may need to re-encrypt the files. I don't see a way around that,
but maybe we could use the scheme with master key somehow.

>> Do we change OIDs of relations when creating the database? If not, we
>> don't need to re-encrypt because having copies of the same block
>> encrypted with the same nonce is not an issue (just like copying
>> encrypted files is not an issue).
>
>Are you thinking that the files can be decrypted by the two keys
>somehow?
>

No, I was kinda assuming the database will start with the same key, but
that might have been a silly idea.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 7/9/19 5:42 PM, Tomas Vondra wrote:
> There are two basic ways to construct nonces - CSPRNG and sequences, and
> then a combination of both, i.e. one part is generated from a sequence
> and one randomly.
>
> FWIW not sure using OIDs as nonces directly is a good idea, as those are
> inherently low entropy data - how often do you see databases with OIDs
> above 1M or so? Probably not very often, and in most cases those are
> databases where those OIDs are for OIDs and large objects, so irrelevant
> for this purpose. I might be wrong but having a 96-bit nonce with maybe
> just 32bits of entrophy seems suspicious.
>
> That does not mean we can't use the OIDs at all, but maybe hashing them
> into a single 4B value, and then picking the remaining 8B randomly.
> Also, we have a "natural" sequence in the database - LSNs, maybe that
> would be a good source of nonces too?

I think you missed the quoted part (upthread) from the NIST document:

  "There are two recommended methods for generating unpredictable IVs.
   The first method is to apply the forward cipher  function, under the
   same key that is used for the encryption of the plaintext, to a
   nonce. The nonce must be a data block that is unique to each
   execution of the encryption operation. For example, the nonce may be
   a counter, as described in Appendix B, or a message number. The
   second method is to generate a random data block using a
   FIPS-approved random number generator."

That first method says a counter as input produces an acceptably
unpredictable IV as long as it is unique to each encryption operation.
If each page is going to be an "encryption operation", so as long as our
input nonce is unique for a given key, we should be ok. If the input
nonce is tableoid+pagenum and the key is different per database (at
least, hopefully different per tablespace too), we should be good to go,
at least from what I can see.

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Greetings,

* Joe Conway (mail@joeconway.com) wrote:
> On 7/9/19 5:42 PM, Tomas Vondra wrote:
> > There are two basic ways to construct nonces - CSPRNG and sequences, and
> > then a combination of both, i.e. one part is generated from a sequence
> > and one randomly.
> >
> > FWIW not sure using OIDs as nonces directly is a good idea, as those are
> > inherently low entropy data - how often do you see databases with OIDs
> > above 1M or so? Probably not very often, and in most cases those are
> > databases where those OIDs are for OIDs and large objects, so irrelevant
> > for this purpose. I might be wrong but having a 96-bit nonce with maybe
> > just 32bits of entrophy seems suspicious.
> >
> > That does not mean we can't use the OIDs at all, but maybe hashing them
> > into a single 4B value, and then picking the remaining 8B randomly.
> > Also, we have a "natural" sequence in the database - LSNs, maybe that
> > would be a good source of nonces too?
>
> I think you missed the quoted part (upthread) from the NIST document:
>
>   "There are two recommended methods for generating unpredictable IVs.
>    The first method is to apply the forward cipher  function, under the
>    same key that is used for the encryption of the plaintext, to a
>    nonce. The nonce must be a data block that is unique to each
>    execution of the encryption operation. For example, the nonce may be
>    a counter, as described in Appendix B, or a message number. The
>    second method is to generate a random data block using a
>    FIPS-approved random number generator."
>
> That first method says a counter as input produces an acceptably
> unpredictable IV as long as it is unique to each encryption operation.
> If each page is going to be an "encryption operation", so as long as our
> input nonce is unique for a given key, we should be ok. If the input
> nonce is tableoid+pagenum and the key is different per database (at
> least, hopefully different per tablespace too), we should be good to go,
> at least from what I can see.

What I think Tomas is getting at here is that we don't write a page only
once.

A nonce of tableoid+pagenum will only be unique the first time we write
out that page.  Seems unlikely that we're only going to be writing these
pages once though- what we need is a nonce that's unique for *every
write* of the 8k page, isn't it?  As every write of the page is going to
be encrypting something new.

With sufficient randomness, we can at least be more likely to have a
unique nonce for each 8K write.  Including the LSN seems like it'd be a
possible alternative.

Thanks,

Stephen

Greetings,

* Ryan Lambert (ryan@rustprooflabs.com) wrote:
> > What I think Tomas is getting at here is that we don't write a page only
> > once.
>
> > A nonce of tableoid+pagenum will only be unique the first time we write
> > out that page.  Seems unlikely that we're only going to be writing these
> > pages once though- what we need is a nonce that's unique for *every
> > write* of the 8k page, isn't it?  As every write of the page is going to
> >  be encrypting something new.
>
> > With sufficient randomness, we can at least be more likely to have a
> > unique nonce for each 8K write.  Including the LSN seems like it'd be a
> > possible alternative.
>
> Agreed.  I know little of the inner details about the LSN but what I read
> in [1] sounds encouraging in addition to tableoid + pagenum.
>
> [1] https://www.postgresql.org/docs/current/datatype-pg-lsn.html

Yes, but it's still something that we'd have to store somewhere- the
actual LSN of the page is going to be in the 8K block.

Unless we decide that we can pull the LSN *out* of the 8K block and
store it unencrypted, and then store the *rest* of the block
encrypted...  That might also allow things like backup software to work
on these encrypted data files for page-level backups without needing
access to the key and that'd be pretty neat.

Of course, as with anything, the more data you expose, the higher the
overall risk that someone can figure out some meaning from it.  Still,
if the idea was that we'd use the LSN in this way, then it'd need to be
stored unencrypted regardless...

Thanks,

Stephen

Greetings,

* Ryan Lambert (ryan@rustprooflabs.com) wrote:
> If a random number were generated instead its result would need to be
> stored somewhere too, correct?

Yes.

Thanks,

Stephen

On Tue, Jul 9, 2019 at 9:01 PM Joe Conway <mail@joeconway.com> wrote:
>
> On 7/9/19 6:07 AM, Peter Eisentraut wrote:
> > On 2019-07-08 18:09, Joe Conway wrote:
> >> In my mind, and in practice to a
> >> large extent, a postgres tablespace == a unique mount point.
> >
> > But a critical difference is that in file systems, a separate mount
> > point has its own journal.
>
> While it would be ideal to have separate WAL, and even separate shared
> buffer pools, per tablespace, I think that is too much complexity for
> the first implementation and we could have a single separate key for all
> WAL for now.

If we encrypt different tables with different keys I think we need to
encrypt WAL with the same keys as we used for tables, as per
discussion so far. And we would need to encrypt each WAL records, not
whole WAL 8k pages.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, Jul 9, 2019 at 10:16 PM Joe Conway <mail@joeconway.com> wrote:
>
> On 7/9/19 8:39 AM, Ryan Lambert wrote:
> > Hi Thomas,
> >
> >> CBC mode does require
> >> random nonces, other modes may be fine with even sequences as long as
> >> the values are not reused.
> >
> > I disagree that CBC mode requires random nonces, at least based on what
> > NIST has published.  They only require that the IV (not the nonce) must
> > be unpredictable per [1]:
> >
> > " For the CBC and CFB modes, the IVs must be unpredictable."
> >
> > The unpredictable IV can be generated from a non-random nonce including
> > a counter:
> >
> > "There are two recommended methods for generating unpredictable IVs. The
> > first method is to apply the forward cipher function, under the same key
> > that is used for the encryption of the plaintext, to a nonce. The nonce
> > must be a data block that is unique to each execution of the encryption
> > operation. For example, the nonce may be a counter, as described in
> > Appendix B, or a message number."
> >
> > [1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
>
>
> The terms nonce and IV are often used more-or-less interchangeably, and
> it is important to be clear when we are talking about an IV specifically
> - an IV is a specific type of nonce. Nonce means "number used once".
> i.e. unique, whereas an IV (for CBC use anyway) should be unique and
> random but not necessarily kept secret.

FWIW, it seems that predictable IVs can sometimes be harmful. See


https://crypto.stackexchange.com/questions/3499/why-cant-the-iv-be-predictable-when-its-said-it-doesnt-need-to-be-a-secret

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Jul 10, 2019 at 11:06 AM Stephen Frost <sfrost@snowman.net> wrote:
>
> Greetings,
>
> * Ryan Lambert (ryan@rustprooflabs.com) wrote:
> > > What I think Tomas is getting at here is that we don't write a page only
> > > once.
> >
> > > A nonce of tableoid+pagenum will only be unique the first time we write
> > > out that page.  Seems unlikely that we're only going to be writing these
> > > pages once though- what we need is a nonce that's unique for *every
> > > write* of the 8k page, isn't it?  As every write of the page is going to
> > >  be encrypting something new.
> >
> > > With sufficient randomness, we can at least be more likely to have a
> > > unique nonce for each 8K write.  Including the LSN seems like it'd be a
> > > possible alternative.
> >
> > Agreed.  I know little of the inner details about the LSN but what I read
> > in [1] sounds encouraging in addition to tableoid + pagenum.
> >
> > [1] https://www.postgresql.org/docs/current/datatype-pg-lsn.html
>
> Yes, but it's still something that we'd have to store somewhere- the
> actual LSN of the page is going to be in the 8K block.

Can we use CBC-ESSIV[1] or XTS[2] instead? IIUC with these modes we
can use table oid and page number for IV or tweak and we don't need to
change them each time to encrypt pages.

[1] https://en.wikipedia.org/wiki/Disk_encryption_theory#Encrypted_salt-sector_initialization_vector_.28ESSIV.29
[2]
https://en.wikipedia.org/wiki/Disk_encryption_theory#XEX-based_tweaked-codebook_mode_with_ciphertext_stealing_(XTS)

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Joe Conway <mail@joeconway.com> wrote:

> On 7/8/19 6:04 PM, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> >> Uh, well, renaming the user was a big problem, but that is the only case
> >> I can think of.  I don't see that as an issue for block or WAL sequence
> >> numbers.  If we want to use a different nonce, we have to find a way to
> >> store it or look it up efficiently.  Considering the nonce size, I don't
> >> see how that is possible.
> >
> > No, this also meant that, as an attacker, I *knew* the salt ahead of
> > time and therefore could build rainbow tables specifically for that
> > salt.  I could also use those *same* tables for any system where that
> > user had an account, even if they used different passwords on different
> > systems...
> >
> > I appreciate that *some* of this might not be completely relevant for
> > the way a nonce is used in cryptography, but I'd be very surprised to
> > have a cryptographer tell me that a deterministic nonce didn't have
> > similar issues or didn't reduce the value of the nonce significantly.
>
> I have worked side by side on projects with bona fide cryptographers and
> I can assure you that they recommended random nonces. Granted, that was
> in the early 2000s, but I don't think "modern cryptography" has changed
> that any more than "web scale" has made Postgres irrelevant in the
> intervening years.

I think that particular threads have to be considered.

> Related links:

> https://defuse.ca/cbcmodeiv.htm
> https://www.cryptofails.com/post/70059609995/crypto-noobs-1-initialization-vectors

The first one looks more in-depth than the other one, so I focused on it:

* "Statistical Correlations between IV and Plaintext"

My understanding is that predictability of the IV (in our implementation of
full-instance encryption [1] we derive the IV from RelFileNode combined with
block number) can reveal information about the first encryption block (16
bytes) of the page, i.e. part of the PageHeaderData structure. I don't think
this leaks any valuable data. And starting the 2nd block, the IV is not
predictable because it is cipher text of the previous block.

* "Chosen-Plaintext Attacks"

The question here is whether we expect the OS admin to have access to the
database. In [1] we currently don't (cloud, where DBA has no control over the
storage layer is the main use case), but if it appears to be the requirement,
I believe CBC-ESSIV mode [2] can fix the problem.

Anyway, I'm not sure if this kind of attack can reveal more information than
something about the first block of the page (the page header), since each of
the following blocks uses ciphertext of the previous block as the IV.

* "Altering the IV Before Decryption"

I don't think this attack needs special attention - page checksums should
reveal it.


[1] https://commitfest.postgresql.org/23/2104/
[2] https://en.wikipedia.org/wiki/Disk_encryption_theory#Encrypted_salt-sector_initialization_vector_.28ESSIV.29

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

> On Tue, Jul 09, 2019 at 03:50:39PM -0400, Bruce Momjian wrote:
> >On Tue, Jul 9, 2019 at 02:09:38PM -0400, Joe Conway wrote:
> >> On 7/9/19 11:11 AM, Bruce Momjian wrote:
> >> > Good point about nonce and IV.  I wonder if running the nonce
> >> > through the cipher with the key makes it random enough to use as an
> >> > IV.
> >>
> >> Based on that NIST document it seems so.
> >>
> >> The trick will be to be 100% sure we never reuse a nonce that is used
> >> to produce the IV when using the same key.
> >>
> >> I think the potential to get that wrong (i.e. inadvertently reuse a
> >> nonce) would lead to using the second described method
> >>
> >>   "The second method is to generate a random data block using a
> >>   FIPS-approved random number generator."
> >>
> >> That method is what I am used to seeing. But with the second method
> >> we need to store the IV, with the first we could reproduce it if we
> >> select our initial nonce carefully.
> >>
> >> So thinking out loud, and perhaps you already said this Bruce, but I
> >> guess the input nonce used to generate the IV could be something like
> >> pg_class.oid and blocknum concatenated together with some delimiting
> >> character as long as we guarantee that we generate different keys in
> >> different databases. Then there would be no need to store the IV since
> >> we could reproduce it.
> >
> >Uh, yes, and no.  Yes, we can use the pg_class.oid (since it has to
> >be preserved by pg_upgrade anyway), and the page number.  However,
> >different databases can have the same pg_class.oid/page number
> >combination, so there would be duplication between databases.  Now, you
> >might say let's add the pg_database.oid, but unfortunately, because of
> >the way we file-system-copy files from one database to another during
> >database creation (it doesn't go through shared buffers), we can't use
> >pg_database.oid as part of the nonce.
> >
> >My only idea here is that we actually decrypt/re-encrypted pages as we
> >copy them at the file system level during database creation to match the
> >new pg_database.oid.  This would allow pg_database.oid in the nonce/IV.
> >(I think we will need to modify pg_upgrade to preserve pg_database.oid.)
> >
> >If the nonce/IV is 96 bits, then that is 12 bytes or 3 4-byte values.
> >pg_class.oid is 4 bytes, pg_database.oid is 4 bytes, and that leaves
> >4-bytes for the block number, which gets us to 32TB before the page
> >counter would overflow a 4-byte value, and our max table size is 32TB
> >anyway, so that all works.
> >
> 
> I don't think that works, because that'd mean we're encrypting the same
> page with the same nonce over and over, which means reusing the reuse
> (even if you hash/encrypt it). Or did I miss something?

I found out that it's wrong to use the same key (or (key, IV) pair) to encrypt
different plain texts [1], however this is about *stream cipher*. There should
be some evidence that *block cipher* has similar weakness before we accept
another restriction on the IV setup.

[1] https://en.wikipedia.org/wiki/Stream_cipher_attacks#Reused_key_attack

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

On 7/9/19 7:28 PM, Stephen Frost wrote:
> Greetings,
>
> * Joe Conway (mail@joeconway.com) wrote:
>> On 7/9/19 5:42 PM, Tomas Vondra wrote:
>> > There are two basic ways to construct nonces - CSPRNG and sequences, and
>> > then a combination of both, i.e. one part is generated from a sequence
>> > and one randomly.
>> >
>> > FWIW not sure using OIDs as nonces directly is a good idea, as those are
>> > inherently low entropy data - how often do you see databases with OIDs
>> > above 1M or so? Probably not very often, and in most cases those are
>> > databases where those OIDs are for OIDs and large objects, so irrelevant
>> > for this purpose. I might be wrong but having a 96-bit nonce with maybe
>> > just 32bits of entrophy seems suspicious.
>> >
>> > That does not mean we can't use the OIDs at all, but maybe hashing them
>> > into a single 4B value, and then picking the remaining 8B randomly.
>> > Also, we have a "natural" sequence in the database - LSNs, maybe that
>> > would be a good source of nonces too?
>>
>> I think you missed the quoted part (upthread) from the NIST document:
>>
>>   "There are two recommended methods for generating unpredictable IVs.
>>    The first method is to apply the forward cipher  function, under the
>>    same key that is used for the encryption of the plaintext, to a
>>    nonce. The nonce must be a data block that is unique to each
>>    execution of the encryption operation. For example, the nonce may be
>>    a counter, as described in Appendix B, or a message number. The
>>    second method is to generate a random data block using a
>>    FIPS-approved random number generator."
>>
>> That first method says a counter as input produces an acceptably
>> unpredictable IV as long as it is unique to each encryption operation.
>> If each page is going to be an "encryption operation", so as long as our
>> input nonce is unique for a given key, we should be ok. If the input
>> nonce is tableoid+pagenum and the key is different per database (at
>> least, hopefully different per tablespace too), we should be good to go,
>> at least from what I can see.
>
> What I think Tomas is getting at here is that we don't write a page only
> once.
>
> A nonce of tableoid+pagenum will only be unique the first time we write
> out that page.  Seems unlikely that we're only going to be writing these
> pages once though- what we need is a nonce that's unique for *every
> write* of the 8k page, isn't it?  As every write of the page is going to
> be encrypting something new.


Hmm, good point. I'm not entirely sure it would be required if the two
page versions don't exist at the same time, but I guess backups mean
that it would, so yeah.

> With sufficient randomness, we can at least be more likely to have a
> unique nonce for each 8K write.  Including the LSN seems like it'd be a
> possible alternative.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/9/19 10:06 PM, Stephen Frost wrote:
> Greetings,
>
> * Ryan Lambert (ryan@rustprooflabs.com) wrote:
>> > What I think Tomas is getting at here is that we don't write a page only
>> > once.
>>
>> > A nonce of tableoid+pagenum will only be unique the first time we write
>> > out that page.  Seems unlikely that we're only going to be writing these
>> > pages once though- what we need is a nonce that's unique for *every
>> > write* of the 8k page, isn't it?  As every write of the page is going to
>> >  be encrypting something new.
>>
>> > With sufficient randomness, we can at least be more likely to have a
>> > unique nonce for each 8K write.  Including the LSN seems like it'd be a
>> > possible alternative.
>>
>> Agreed.  I know little of the inner details about the LSN but what I read
>> in [1] sounds encouraging in addition to tableoid + pagenum.
>>
>> [1] https://www.postgresql.org/docs/current/datatype-pg-lsn.html
>
> Yes, but it's still something that we'd have to store somewhere- the
> actual LSN of the page is going to be in the 8K block.
>
> Unless we decide that we can pull the LSN *out* of the 8K block and
> store it unencrypted, and then store the *rest* of the block
> encrypted...  That might also allow things like backup software to work
> on these encrypted data files for page-level backups without needing
> access to the key and that'd be pretty neat.
>
> Of course, as with anything, the more data you expose, the higher the
> overall risk that someone can figure out some meaning from it.  Still,
> if the idea was that we'd use the LSN in this way, then it'd need to be
> stored unencrypted regardless...

I don't think we are going to be able to eliminate every possible
side-channel anyway -- this seems like a good compromise to me.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/10/19 2:40 AM, Masahiko Sawada wrote:
> On Tue, Jul 9, 2019 at 10:16 PM Joe Conway <mail@joeconway.com> wrote:
>>
>> On 7/9/19 8:39 AM, Ryan Lambert wrote:
>> > Hi Thomas,
>> >
>> >> CBC mode does require
>> >> random nonces, other modes may be fine with even sequences as long as
>> >> the values are not reused.
>> >
>> > I disagree that CBC mode requires random nonces, at least based on what
>> > NIST has published.  They only require that the IV (not the nonce) must
>> > be unpredictable per [1]:
>> >
>> > " For the CBC and CFB modes, the IVs must be unpredictable."
>> >
>> > The unpredictable IV can be generated from a non-random nonce including
>> > a counter:
>> >
>> > "There are two recommended methods for generating unpredictable IVs. The
>> > first method is to apply the forward cipher function, under the same key
>> > that is used for the encryption of the plaintext, to a nonce. The nonce
>> > must be a data block that is unique to each execution of the encryption
>> > operation. For example, the nonce may be a counter, as described in
>> > Appendix B, or a message number."
>> >
>> > [1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
>>
>>
>> The terms nonce and IV are often used more-or-less interchangeably, and
>> it is important to be clear when we are talking about an IV specifically
>> - an IV is a specific type of nonce. Nonce means "number used once".
>> i.e. unique, whereas an IV (for CBC use anyway) should be unique and
>> random but not necessarily kept secret.
>
> FWIW, it seems that predictable IVs can sometimes be harmful. See


Yes, for CBC as I said above "IV ... should be unique and random but not
necessarily kept secret". You can argue if the word "random" should read
"unpredictable" instead, but that was the intention.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/10/19 2:38 AM, Masahiko Sawada wrote:
> On Tue, Jul 9, 2019 at 9:01 PM Joe Conway <mail@joeconway.com> wrote:
>>
>> On 7/9/19 6:07 AM, Peter Eisentraut wrote:
>> > On 2019-07-08 18:09, Joe Conway wrote:
>> >> In my mind, and in practice to a
>> >> large extent, a postgres tablespace == a unique mount point.
>> >
>> > But a critical difference is that in file systems, a separate mount
>> > point has its own journal.
>>
>> While it would be ideal to have separate WAL, and even separate shared
>> buffer pools, per tablespace, I think that is too much complexity for
>> the first implementation and we could have a single separate key for all
>> WAL for now.
>
> If we encrypt different tables with different keys I think we need to
> encrypt WAL with the same keys as we used for tables, as per
> discussion so far. And we would need to encrypt each WAL records, not
> whole WAL 8k pages.

That is not a technical requirement to be sure. We may decide we want
that from a security perspective, but that point is debatable. There
have been different goals expressed on this thread:

1. Keep user 1 from decrypting data A and user 2 from decrypting data B
2. Limit the amount of data encrypted with key Kn

We can use K1 for A, K2 for B, and K3 for WAL and achieve goal #2. As
Stephen pointed out, goal #1 would be great to have, but I am not sure
there is consensus that it is required, at least not for the initial
implementation.

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/10/19 4:47 AM, Antonin Houska wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>> I don't think that works, because that'd mean we're encrypting the same
>> page with the same nonce over and over, which means reusing the reuse
>> (even if you hash/encrypt it). Or did I miss something?
>
> I found out that it's wrong to use the same key (or (key, IV) pair) to encrypt
> different plain texts [1], however this is about *stream cipher*. There should
> be some evidence that *block cipher* has similar weakness before we accept
> another restriction on the IV setup.
>
> [1] https://en.wikipedia.org/wiki/Stream_cipher_attacks#Reused_key_attack

There is plenty of guidance that specifies CBC requires unique,
unpredictable, but not necessarily secret IV. See for example Appendix C
here:

https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/10/19 4:24 AM, Antonin Houska wrote:
> Joe Conway <mail@joeconway.com> wrote:
>
>> On 7/8/19 6:04 PM, Stephen Frost wrote:
>> > * Bruce Momjian (bruce@momjian.us) wrote:
>> >> Uh, well, renaming the user was a big problem, but that is the only case
>> >> I can think of.  I don't see that as an issue for block or WAL sequence
>> >> numbers.  If we want to use a different nonce, we have to find a way to
>> >> store it or look it up efficiently.  Considering the nonce size, I don't
>> >> see how that is possible.
>> >
>> > No, this also meant that, as an attacker, I *knew* the salt ahead of
>> > time and therefore could build rainbow tables specifically for that
>> > salt.  I could also use those *same* tables for any system where that
>> > user had an account, even if they used different passwords on different
>> > systems...
>> >
>> > I appreciate that *some* of this might not be completely relevant for
>> > the way a nonce is used in cryptography, but I'd be very surprised to
>> > have a cryptographer tell me that a deterministic nonce didn't have
>> > similar issues or didn't reduce the value of the nonce significantly.
>>
>> I have worked side by side on projects with bona fide cryptographers and
>> I can assure you that they recommended random nonces. Granted, that was
>> in the early 2000s, but I don't think "modern cryptography" has changed
>> that any more than "web scale" has made Postgres irrelevant in the
>> intervening years.
>
> I think that particular threads have to be considered.
>
>> Related links:
>
>> https://defuse.ca/cbcmodeiv.htm
>> https://www.cryptofails.com/post/70059609995/crypto-noobs-1-initialization-vectors
>
> The first one looks more in-depth than the other one, so I focused on it:
>
> * "Statistical Correlations between IV and Plaintext"
>
> My understanding is that predictability of the IV (in our implementation of
> full-instance encryption [1] we derive the IV from RelFileNode combined with
> block number) can reveal information about the first encryption block (16
> bytes) of the page, i.e. part of the PageHeaderData structure. I don't think
> this leaks any valuable data. And starting the 2nd block, the IV is not
> predictable because it is cipher text of the previous block.
>
> * "Chosen-Plaintext Attacks"
>
> The question here is whether we expect the OS admin to have access to the
> database. In [1] we currently don't (cloud, where DBA has no control over the
> storage layer is the main use case), but if it appears to be the requirement,
> I believe CBC-ESSIV mode [2] can fix the problem.
>
> Anyway, I'm not sure if this kind of attack can reveal more information than
> something about the first block of the page (the page header), since each of
> the following blocks uses ciphertext of the previous block as the IV.
>
> * "Altering the IV Before Decryption"
>
> I don't think this attack needs special attention - page checksums should
> reveal it.
>
>
> [1] https://commitfest.postgresql.org/23/2104/
> [2] https://en.wikipedia.org/wiki/Disk_encryption_theory#Encrypted_salt-sector_initialization_vector_.28ESSIV.29
>

Please see my other reply (and
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
appendix C as pointed out by Ryan downthread).

At least in my mind, I trust a published specification from the
nation-state level over random blogs or wikipedia. If we can find some
equivalent published standards that contradict NIST we should discuss
it, but for my money I would prefer to stick with the NIST recommended
method to produce the IVs.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Greetings,

* Joe Conway (mail@joeconway.com) wrote:
> On 7/9/19 7:28 PM, Stephen Frost wrote:
> > * Joe Conway (mail@joeconway.com) wrote:
> >> On 7/9/19 5:42 PM, Tomas Vondra wrote:
> >> > There are two basic ways to construct nonces - CSPRNG and sequences, and
> >> > then a combination of both, i.e. one part is generated from a sequence
> >> > and one randomly.
> >> >
> >> > FWIW not sure using OIDs as nonces directly is a good idea, as those are
> >> > inherently low entropy data - how often do you see databases with OIDs
> >> > above 1M or so? Probably not very often, and in most cases those are
> >> > databases where those OIDs are for OIDs and large objects, so irrelevant
> >> > for this purpose. I might be wrong but having a 96-bit nonce with maybe
> >> > just 32bits of entrophy seems suspicious.
> >> >
> >> > That does not mean we can't use the OIDs at all, but maybe hashing them
> >> > into a single 4B value, and then picking the remaining 8B randomly.
> >> > Also, we have a "natural" sequence in the database - LSNs, maybe that
> >> > would be a good source of nonces too?
> >>
> >> I think you missed the quoted part (upthread) from the NIST document:
> >>
> >>   "There are two recommended methods for generating unpredictable IVs.
> >>    The first method is to apply the forward cipher  function, under the
> >>    same key that is used for the encryption of the plaintext, to a
> >>    nonce. The nonce must be a data block that is unique to each
> >>    execution of the encryption operation. For example, the nonce may be
> >>    a counter, as described in Appendix B, or a message number. The
> >>    second method is to generate a random data block using a
> >>    FIPS-approved random number generator."
> >>
> >> That first method says a counter as input produces an acceptably
> >> unpredictable IV as long as it is unique to each encryption operation.
> >> If each page is going to be an "encryption operation", so as long as our
> >> input nonce is unique for a given key, we should be ok. If the input
> >> nonce is tableoid+pagenum and the key is different per database (at
> >> least, hopefully different per tablespace too), we should be good to go,
> >> at least from what I can see.
> >
> > What I think Tomas is getting at here is that we don't write a page only
> > once.
> >
> > A nonce of tableoid+pagenum will only be unique the first time we write
> > out that page.  Seems unlikely that we're only going to be writing these
> > pages once though- what we need is a nonce that's unique for *every
> > write* of the 8k page, isn't it?  As every write of the page is going to
> > be encrypting something new.
>
> Hmm, good point. I'm not entirely sure it would be required if the two
> page versions don't exist at the same time, but I guess backups mean
> that it would, so yeah.

Uh, or an attacker got a copy of the page and then just waited a few
minutes for a new version to be written and then grabbed that...

Definitely not limited to just concerns about the fact that other
versions would exist in backups too.

Thanks,

Stephen

On 7/10/19 2:45 AM, Masahiko Sawada wrote:
> On Wed, Jul 10, 2019 at 11:06 AM Stephen Frost <sfrost@snowman.net> wrote:
>>
>> Greetings,
>>
>> * Ryan Lambert (ryan@rustprooflabs.com) wrote:
>> > > What I think Tomas is getting at here is that we don't write a page only
>> > > once.
>> >
>> > > A nonce of tableoid+pagenum will only be unique the first time we write
>> > > out that page.  Seems unlikely that we're only going to be writing these
>> > > pages once though- what we need is a nonce that's unique for *every
>> > > write* of the 8k page, isn't it?  As every write of the page is going to
>> > >  be encrypting something new.
>> >
>> > > With sufficient randomness, we can at least be more likely to have a
>> > > unique nonce for each 8K write.  Including the LSN seems like it'd be a
>> > > possible alternative.
>> >
>> > Agreed.  I know little of the inner details about the LSN but what I read
>> > in [1] sounds encouraging in addition to tableoid + pagenum.
>> >
>> > [1] https://www.postgresql.org/docs/current/datatype-pg-lsn.html
>>
>> Yes, but it's still something that we'd have to store somewhere- the
>> actual LSN of the page is going to be in the 8K block.
>
> Can we use CBC-ESSIV[1] or XTS[2] instead? IIUC with these modes we
> can use table oid and page number for IV or tweak and we don't need to
> change them each time to encrypt pages.
>
> [1] https://en.wikipedia.org/wiki/Disk_encryption_theory#Encrypted_salt-sector_initialization_vector_.28ESSIV.29
> [2]
https://en.wikipedia.org/wiki/Disk_encryption_theory#XEX-based_tweaked-codebook_mode_with_ciphertext_stealing_(XTS)


From what I can tell [1] is morally equivalent to the NIST method and
does nothing to change the fact that the input nonce needs to be unique
for each encryption operation. I have not had time to review [2] yet...

While it would be very tempting to convince ourselves that a unique
input nonce is not a requirement, I think we are better off being
conservative unless we find some extremely clear guidance that allows us
to draw that conclusion.

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/10/19 8:34 AM, Stephen Frost wrote:
> Greetings,
>
> * Joe Conway (mail@joeconway.com) wrote:
>> On 7/9/19 7:28 PM, Stephen Frost wrote:
>> > * Joe Conway (mail@joeconway.com) wrote:
>> >> On 7/9/19 5:42 PM, Tomas Vondra wrote:
>> >> > There are two basic ways to construct nonces - CSPRNG and sequences, and
>> >> > then a combination of both, i.e. one part is generated from a sequence
>> >> > and one randomly.
>> >> >
>> >> > FWIW not sure using OIDs as nonces directly is a good idea, as those are
>> >> > inherently low entropy data - how often do you see databases with OIDs
>> >> > above 1M or so? Probably not very often, and in most cases those are
>> >> > databases where those OIDs are for OIDs and large objects, so irrelevant
>> >> > for this purpose. I might be wrong but having a 96-bit nonce with maybe
>> >> > just 32bits of entrophy seems suspicious.
>> >> >
>> >> > That does not mean we can't use the OIDs at all, but maybe hashing them
>> >> > into a single 4B value, and then picking the remaining 8B randomly.
>> >> > Also, we have a "natural" sequence in the database - LSNs, maybe that
>> >> > would be a good source of nonces too?
>> >>
>> >> I think you missed the quoted part (upthread) from the NIST document:
>> >>
>> >>   "There are two recommended methods for generating unpredictable IVs.
>> >>    The first method is to apply the forward cipher  function, under the
>> >>    same key that is used for the encryption of the plaintext, to a
>> >>    nonce. The nonce must be a data block that is unique to each
>> >>    execution of the encryption operation. For example, the nonce may be
>> >>    a counter, as described in Appendix B, or a message number. The
>> >>    second method is to generate a random data block using a
>> >>    FIPS-approved random number generator."
>> >>
>> >> That first method says a counter as input produces an acceptably
>> >> unpredictable IV as long as it is unique to each encryption operation.
>> >> If each page is going to be an "encryption operation", so as long as our
>> >> input nonce is unique for a given key, we should be ok. If the input
>> >> nonce is tableoid+pagenum and the key is different per database (at
>> >> least, hopefully different per tablespace too), we should be good to go,
>> >> at least from what I can see.
>> >
>> > What I think Tomas is getting at here is that we don't write a page only
>> > once.
>> >
>> > A nonce of tableoid+pagenum will only be unique the first time we write
>> > out that page.  Seems unlikely that we're only going to be writing these
>> > pages once though- what we need is a nonce that's unique for *every
>> > write* of the 8k page, isn't it?  As every write of the page is going to
>> > be encrypting something new.
>>
>> Hmm, good point. I'm not entirely sure it would be required if the two
>> page versions don't exist at the same time, but I guess backups mean
>> that it would, so yeah.
>
> Uh, or an attacker got a copy of the page and then just waited a few
> minutes for a new version to be written and then grabbed that...
>
> Definitely not limited to just concerns about the fact that other
> versions would exist in backups too.

Agreed :-/

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Tue, Jul 09, 2019 at 10:06:33PM -0400, Stephen Frost wrote:
>Greetings,
>
>* Ryan Lambert (ryan@rustprooflabs.com) wrote:
>> > What I think Tomas is getting at here is that we don't write a page only
>> > once.
>>

Yes, that's what I meant.

>> > A nonce of tableoid+pagenum will only be unique the first time we write
>> > out that page.  Seems unlikely that we're only going to be writing these
>> > pages once though- what we need is a nonce that's unique for *every
>> > write* of the 8k page, isn't it?  As every write of the page is going to
>> >  be encrypting something new.
>>
>> > With sufficient randomness, we can at least be more likely to have a
>> > unique nonce for each 8K write.  Including the LSN seems like it'd be a
>> > possible alternative.
>>
>> Agreed.  I know little of the inner details about the LSN but what I read
>> in [1] sounds encouraging in addition to tableoid + pagenum.
>>
>> [1] https://www.postgresql.org/docs/current/datatype-pg-lsn.html
>
>Yes, but it's still something that we'd have to store somewhere- the
>actual LSN of the page is going to be in the 8K block.
>
>Unless we decide that we can pull the LSN *out* of the 8K block and
>store it unencrypted, and then store the *rest* of the block
>encrypted...  That might also allow things like backup software to work
>on these encrypted data files for page-level backups without needing
>access to the key and that'd be pretty neat.
>
>Of course, as with anything, the more data you expose, the higher the
>overall risk that someone can figure out some meaning from it.  Still,
>if the idea was that we'd use the LSN in this way, then it'd need to be
>stored unencrypted regardless...
>

Elsewhere in this thread I've already proposed to leave a bit of space at
the end of a page unencrypted, with page-level encryption metadata. That
might be the nonce (no matter how we end up computing it), key ID used to
encrypt this page, etc.

I don't think we need to put the whole LSN into the nonce in plaintext.
What I was imagining was intead using something like

    sha2(LSN, oid, blockno, random())

or something like that.

Of course, having the LSN (and other stuff like page checksum) unencrypted
would be pretty useful - as you note.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jul 10, 2019 at 03:38:54PM +0900, Masahiko Sawada wrote:
>On Tue, Jul 9, 2019 at 9:01 PM Joe Conway <mail@joeconway.com> wrote:
>>
>> On 7/9/19 6:07 AM, Peter Eisentraut wrote:
>> > On 2019-07-08 18:09, Joe Conway wrote:
>> >> In my mind, and in practice to a
>> >> large extent, a postgres tablespace == a unique mount point.
>> >
>> > But a critical difference is that in file systems, a separate mount
>> > point has its own journal.
>>
>> While it would be ideal to have separate WAL, and even separate shared
>> buffer pools, per tablespace, I think that is too much complexity for
>> the first implementation and we could have a single separate key for all
>> WAL for now.
>
>If we encrypt different tables with different keys I think we need to
>encrypt WAL with the same keys as we used for tables, as per
>discussion so far. And we would need to encrypt each WAL records, not
>whole WAL 8k pages.
>

I don't think we actually need that - we need to ensure that users don't
have access to the key used to encrypt WAL.

This is very much a question of the threat model. If we see TDE as a
data-at-rest solution, then I think it's fine to have a separate keyring
with such keys, and only allow access to system processes.

If our thread model includes cases where people can read memory (does not
matter if it's because of a vulnerability, privilege escalation or just
allowing the people to load an extension with custom C function), then I
think we've already lost. Those people will be able to read the keys
anyway, no matter how many keys are used to encrypt the WAL.

We may need to change how WAL writing works, so that individual backends
don't really write into the WAL buffers directly, and intead hand-over the
data to a separate process (with access to the key). We already have the
walwriter, but IIRC it may not be running and we consider that to be OK.
Or maybe we could have "encrypter" process that does just that.

That's surely non-trivial work, but it seems like much less work compared
to reworking the WAL format to allow WAL to be encrypted with different
keys. At least for v0 that should be OK.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote:
> Please see my other reply (and
> https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> appendix C as pointed out by Ryan downthread).
> 
> At least in my mind, I trust a published specification from the
> nation-state level over random blogs or wikipedia. If we can find some
> equivalent published standards that contradict NIST we should discuss
> it, but for my money I would prefer to stick with the NIST recommended
> method to produce the IVs.

So, we have had a flurry of activity on this thread in the past day, so
let me summarize:

*  Using the database oid does make the nonce unique in the cluster as
long as we re-encrypt when we do CREATE DATABASE.  We can avoid some of
that by not encrypting template1, but we have the WITH TEMPLATE option
to use other databases as templates, so we might as well always just
decrypt/re-encrypted.

*  However, the page will be rewritten many times, so if we use just
pg_database/pg_class/page-offset for the nonce, we are re-encrypting
with the same nonce/IV for multiple page values, which is a security
issue.

*  Using the LSN as part of the nonce fixes both problems, and has a
   third benefit:

    *  We don't need to decrypt/re-encrypt during CREATE DATABASE since
       the page contents are the same in both places, and once one
       database changes its pages, it gets a new LSN, and hence a new
       nonce/IV.

    *  For each change of an 8k page, you get a new nonce/IV, so you
       are not encrypting different data with the same nonce/IV

    *  This avoids requiring pg_upgrade to preserve database oids.

*  It was determined that running known values like pg_class.oid, LSN,
page-number to create a nonce and running those through the encryption function
to create an IV is sufficient.

However, the LSN must then be visible on the encrypted pages.  I would
like to avoid having different page formats for encrypted and
non-encrypted pages, because if we require additional storage for
encrypted pages (like adding a random number), existing non-encrypted
pages might not be able to fit in the encrypted format, causing
complexity when accessing them and when converting tables to encrypted
format.

Looking at the page header, I see:

    typedef struct PageHeaderData
    {
        /* XXX LSN is member of *any* block, not only page-organized ones */
        PageXLogRecPtr pd_lsn;      /* LSN: next byte after last byte of xlog
                                     * record for last change to this page */
        uint16      pd_checksum;    /* checksum */
        uint16      pd_flags;       /* flag bits, see below */
        LocationIndex pd_lower;     /* offset to start of free space */
        LocationIndex pd_upper;     /* offset to end of free space */

pd_lsn/PageXLogRecPtr is 8 bytes.  (We might only want to use the low
order bits for the nonce.)  LocationIndex is 16 bits, meaning that the
four fields listed above are 16-bytes wide, which is the width of the
typical AES cipher mode block.  I suggest we _not_ encrypt the first 16
bytes of each 8k page, and start encrypting at byte 17 --- that way,
these values are visible and can be used as part of the nonce to create
an IV.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-10, Bruce Momjian wrote:

> *  Using the LSN as part of the nonce fixes both problems, and has a
>    third benefit:
> 
>     *  We don't need to decrypt/re-encrypt during CREATE DATABASE since
>        the page contents are the same in both places, and once one
>        database changes its pages, it gets a new LSN, and hence a new
>        nonce/IV.
> 
>     *  For each change of an 8k page, you get a new nonce/IV, so you
>        are not encrypting different data with the same nonce/IV
> 
>     *  This avoids requiring pg_upgrade to preserve database oids.

An ignorant question -- what is it that gets stored in the page for
decryption use, the nonce or the IV derived from it?  I think if you
want to store the nonce, you'd have to store the database OID, because
otherwise how do you know which database OID to use to determine the
full nonce after cloning a database?  You already have the table OID in
the catalog and the LSN in the page header, so you're only missing the
database OID.  (Assuming you make the nonce be database OID || relation
OID || page LSN)

Also, how are key changes handled?  Do we need to store a key identifier
in each page?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jul 10, 2019 at 01:04:47PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-10, Bruce Momjian wrote:
> 
> > *  Using the LSN as part of the nonce fixes both problems, and has a
> >    third benefit:
> > 
> >     *  We don't need to decrypt/re-encrypt during CREATE DATABASE since
> >        the page contents are the same in both places, and once one
> >        database changes its pages, it gets a new LSN, and hence a new
> >        nonce/IV.
> > 
> >     *  For each change of an 8k page, you get a new nonce/IV, so you
> >        are not encrypting different data with the same nonce/IV
> > 
> >     *  This avoids requiring pg_upgrade to preserve database oids.
> 
> An ignorant question -- what is it that gets stored in the page for
> decryption use, the nonce or the IV derived from it?  I think if you
> want to store the nonce, you'd have to store the database OID, because
> otherwise how do you know which database OID to use to determine the
> full nonce after cloning a database?  You already have the table OID in
> the catalog and the LSN in the page header, so you're only missing the
> database OID.  (Assuming you make the nonce be database OID || relation
> OID || page LSN)

You are right that if you used the database oid in the nonce, you would
need to decrypt/re-encrypt the data during CREATE DATABASE, or store
the original database oid in the page.

The new approach is that a single key would be used for all databases
and the WAL, and use the LSN instead of the database oid, so there is no
need to know which database originally encrypted the page --- any
database can decrypt it.

> Also, how are key changes handled?  Do we need to store a key identifier
> in each page?

Uh, we have not started discussing that yet.  I am thinking we might
need to store the key identifier in the pg_class table and then create a
command to re-encrypt tables.  We can re-key a page at a time, but we
would still need to know when all pages/tables are no longer using the
old key, so doing it just at the table/index level seems appropriate.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Ryan Lambert (ryan@rustprooflabs.com) wrote:
> > what is it that gets stored in the page for
> > decryption use, the nonce or the IV derived from it?
>
> I believe storing the IV is preferable and still secure per [1]: "The IV
> need not be secret"
>
> Beyond needing the database oid, if every decrypt function has to
> regenerate the IV from the nonce that will affect performance.  I don't
> know how expensive the forward hash is but it won't be free.

Compared to the syscall and possible disk i/o required, I'm not sure
that's something we really need to try to optimize for, particularly if
we could store something more generally useful (like the LSN) in that
little bit of space that's available in each page.

Thanks,

Stephen

On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote:
> 
>     what is it that gets stored in the page for
>     decryption use, the nonce or the IV derived from it?
> 
> 
> I believe storing the IV is preferable and still secure per [1]: "The IV need
> not be secret"
> 
> Beyond needing the database oid, if every decrypt function has to regenerate
> the IV from the nonce that will affect performance.  I don't know how expensive
> the forward hash is but it won't be free.

Well, I think we have three options.  We have 3 4-byte integers
(pg_class.oid, LSN, page-number) that could be concatenated to be the
IV, we could run those through a hash, or we could run them through the
encryption function with the secret.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote:
> >
> >     what is it that gets stored in the page for
> >     decryption use, the nonce or the IV derived from it?
> >
> >
> > I believe storing the IV is preferable and still secure per [1]: "The IV need
> > not be secret"
> >
> > Beyond needing the database oid, if every decrypt function has to regenerate
> > the IV from the nonce that will affect performance.  I don't know how expensive
> > the forward hash is but it won't be free.
>
> Well, I think we have three options.  We have 3 4-byte integers
> (pg_class.oid, LSN, page-number) that could be concatenated to be the
> IV, we could run those through a hash, or we could run them through the
> encryption function with the secret.

I didn't see where it was said that using a hash was a good idea in this
context..?  Encrypting it with the key looked like it was discussed as a
viable option.  I had understood that part of the point of using the
table OID and page-number was also so that we didn't have to explicitly
store the result, therefore requiring us to need less space on the page
to make this happen.

Thanks,

Stephen

On Wed, Jul 10, 2019 at 02:44:30PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Wed, Jul 10, 2019 at 12:38:02PM -0600, Ryan Lambert wrote:
> > > 
> > >     what is it that gets stored in the page for
> > >     decryption use, the nonce or the IV derived from it?
> > > 
> > > 
> > > I believe storing the IV is preferable and still secure per [1]: "The IV need
> > > not be secret"
> > > 
> > > Beyond needing the database oid, if every decrypt function has to regenerate
> > > the IV from the nonce that will affect performance.  I don't know how expensive
> > > the forward hash is but it won't be free.
> > 
> > Well, I think we have three options.  We have 3 4-byte integers
> > (pg_class.oid, LSN, page-number) that could be concatenated to be the
> > IV, we could run those through a hash, or we could run them through the
> > encryption function with the secret.
> 
> I didn't see where it was said that using a hash was a good idea in this
> context..?  Encrypting it with the key looked like it was discussed as a

I didn't either, except it was referenced above as "forward hash".  I
don't know why that was suggested, which is why I listed it as an
option/suggestion.

> viable option.  I had understood that part of the point of using the

Agreed.

> table OID and page-number was also so that we didn't have to explicitly
> store the result, therefore requiring us to need less space on the page
> to make this happen.

Yep!

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Ryan Lambert (ryan@rustprooflabs.com) wrote:
> > I didn't either, except it was referenced above as "forward hash".  I
> > don't know why that was suggested, which is why I listed it as an
> > option/suggestion.
>
> My bad, sorry for the confusion!  I meant to say "cipher" not "hash".  I
> was (trying to) refer to the method of generating unpredictable IV from
> nonces using the forward cipher function and the encryption key.
> Too many closely related words with very specific meanings.

No worries, just want to try and be clear on these things..  Too easy to
mistakenly think that doing this very-similar-thing will be as secure as
doing the recommended-thing (particularly when the recommended-thing is
a lot harder...), and we don't want to end up doing that and then
discovering it isn't actually secure..

Thanks!

Stephen

On Wed, Jul 10, 2019 at 02:57:54PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Ryan Lambert (ryan@rustprooflabs.com) wrote:
> > > I didn't either, except it was referenced above as "forward hash".  I
> > > don't know why that was suggested, which is why I listed it as an
> > > option/suggestion.
> > 
> > My bad, sorry for the confusion!  I meant to say "cipher" not "hash".  I
> > was (trying to) refer to the method of generating unpredictable IV from
> > nonces using the forward cipher function and the encryption key.
> > Too many closely related words with very specific meanings.
> 
> No worries, just want to try and be clear on these things..  Too easy to
> mistakenly think that doing this very-similar-thing will be as secure as
> doing the recommended-thing (particularly when the recommended-thing is
> a lot harder...), and we don't want to end up doing that and then
> discovering it isn't actually secure..

Good, so I think we all now agree we have to put the nonce
(pg_class.oid, LSN, page-number) though the cipher using the secret.  I
think Stephen is right that the overhead of this will be minimal for 8k
page writes, and for WAL, we only need to generate the IV when we start
a new 16MB segment, so again, minimal overhead.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-10, Bruce Momjian wrote:

> Good, so I think we all now agree we have to put the nonce
> (pg_class.oid, LSN, page-number) though the cipher using the secret.

Actually, why do you need the page number in the nonce?  The LSN already
distinguishes pages -- you can't have two pages with the same LSN, can
you?  (I do think you can have multiple writes of the same page with
different LSNs, if you change hint bits and don't write WAL about it,
but maybe we should force CRC enabled in encrypted tables, which I think
closes this hole?)

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> On 2019-Jul-10, Bruce Momjian wrote:
>
> > Good, so I think we all now agree we have to put the nonce
> > (pg_class.oid, LSN, page-number) though the cipher using the secret.
>
> Actually, why do you need the page number in the nonce?  The LSN already
> distinguishes pages -- you can't have two pages with the same LSN, can
> you?  (I do think you can have multiple writes of the same page with
> different LSNs, if you change hint bits and don't write WAL about it,
> but maybe we should force CRC enabled in encrypted tables, which I think
> closes this hole?)

The point about the LSN not changing is definitely a very good one..  I
agree that we should require checksums to deal with that possibility.

Thanks,

Stephen

On Wed, Jul 10, 2019 at 03:53:55PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-10, Bruce Momjian wrote:
> 
> > Good, so I think we all now agree we have to put the nonce
> > (pg_class.oid, LSN, page-number) though the cipher using the secret.
> 
> Actually, why do you need the page number in the nonce?  The LSN already
> distinguishes pages -- you can't have two pages with the same LSN, can
> you?  (I do think you can have multiple writes of the same page with
> different LSNs, if you change hint bits and don't write WAL about it,
> but maybe we should force CRC enabled in encrypted tables, which I think
> closes this hole?)

Uh, what if a transaction modifies page 0 and page 1 of the same table
--- don't those pages have the same LSN.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-10, Bruce Momjian wrote:

> Uh, what if a transaction modifies page 0 and page 1 of the same table
> --- don't those pages have the same LSN.

No, because WAL being a physical change log, each page gets its own
WAL record with its own LSN.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7/10/19 3:53 PM, Alvaro Herrera wrote:
> On 2019-Jul-10, Bruce Momjian wrote:
> 
>> Good, so I think we all now agree we have to put the nonce
>> (pg_class.oid, LSN, page-number) though the cipher using the secret.

(been traveling -- just trying to get caught up on this thread)

> Actually, why do you need the page number in the nonce?  The LSN already
> distinguishes pages -- you can't have two pages with the same LSN, can
> you?  (I do think you can have multiple writes of the same page with
> different LSNs, if you change hint bits and don't write WAL about it,

Do you mean "multiple writes of the same page without..."?

> but maybe we should force CRC enabled in encrypted tables, which I think
> closes this hole?)

If we can use the LSN (perhaps with CRC) without the page number that
would seem to be a good idea.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote:
>On 2019-Jul-10, Bruce Momjian wrote:
>
>> Uh, what if a transaction modifies page 0 and page 1 of the same table
>> --- don't those pages have the same LSN.
>
>No, because WAL being a physical change log, each page gets its own
>WAL record with its own LSN.
>

What if you have wal_log_hints=off? AFAIK that won't change the page LSN.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

Greetings,

* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote:
> >On 2019-Jul-10, Bruce Momjian wrote:
> >
> >>Uh, what if a transaction modifies page 0 and page 1 of the same table
> >>--- don't those pages have the same LSN.
> >
> >No, because WAL being a physical change log, each page gets its own
> >WAL record with its own LSN.
> >
>
> What if you have wal_log_hints=off? AFAIK that won't change the page LSN.

Alvaro suggested elsewhere that we require checksums for these, which
would also force wal_log_hints to be on, and therefore the LSN would
change.

Thanks,

Stephen

On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote:
>Greetings,
>
>* Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
>> On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote:
>> >On 2019-Jul-10, Bruce Momjian wrote:
>> >
>> >>Uh, what if a transaction modifies page 0 and page 1 of the same table
>> >>--- don't those pages have the same LSN.
>> >
>> >No, because WAL being a physical change log, each page gets its own
>> >WAL record with its own LSN.
>> >
>>
>> What if you have wal_log_hints=off? AFAIK that won't change the page LSN.
>
>Alvaro suggested elsewhere that we require checksums for these, which
>would also force wal_log_hints to be on, and therefore the LSN would
>change.
>

Oh, I see - yes, that would solve the hint bits issue. Not sure we want
to combine the features like this, though, as it increases the costs of
TDE. But maybe it's the best solution.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 2019-Jul-10, Joe Conway wrote:

> On 7/10/19 3:53 PM, Alvaro Herrera wrote:

> > (I do think you can have multiple writes of the same page with
> > different LSNs, if you change hint bits and don't write WAL about it,
> 
> Do you mean "multiple writes of the same page without..."?

Right, "twice the same page with the same LSN" is what I was thinking,
which is basically the question Tomas asked afterwards.

> > but maybe we should force CRC enabled in encrypted tables, which I think
> > closes this hole?)
> 
> If we can use the LSN (perhaps with CRC) without the page number that
> would seem to be a good idea.

Umm, I'm not advocating using the CRC as part of the nonce, because that
seems a terrible idea.  I was just saying that if you enable CRC, then
even hint bit changes cause LSN changes (and thus IV changes) because of
the necessary FPIs, so you shouldn't get two writes with the same LSN.

With all this said, I think the case for writing two pages with the same
IV is being overstated a little bit.  As I understand, the reason we
want to avoid using the same IV for too many pages is to dodge a
cryptanalysis attack, which requires a large amount of data encrypted
with the same key/IV in order to be effective.  But if we have two
copies of the same page encrypted with the same key/IV, yes it's twice
as much data as just one copy of the page with that key/IV, but it still
seems like a sufficiently low amount of data that cryptanalysis is
unfeasible.  Right?  I mean, webservers send hundreds of kilobytes
encrypted with the same key; they avoid sending megabytes of it with the
same key/IV, but getting too worked up about 16 kB when we think 8 kB is
fine seems over the top.

So I guess the question is how much data is considered sufficient for a
successful, practical cryptanalysis attack?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Jul 10, 2019 at 06:28:42PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-10, Joe Conway wrote:
> 
> > On 7/10/19 3:53 PM, Alvaro Herrera wrote:
> 
> > > (I do think you can have multiple writes of the same page with
> > > different LSNs, if you change hint bits and don't write WAL about it,
> > 
> > Do you mean "multiple writes of the same page without..."?
> 
> Right, "twice the same page with the same LSN" is what I was thinking,
> which is basically the question Tomas asked afterwards.

Just to clarify, the case being discussed above is where we modify a
page with a new row, write the page with an LSN, then change a hint bit
on the page, and, if log_hint_bits is false, write the page with the
hint bit change without changing the LSN since we didn't log hint bits.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 11, 2019 at 12:18:47AM +0200, Tomas Vondra wrote:
> On Wed, Jul 10, 2019 at 06:04:30PM -0400, Stephen Frost wrote:
> > Greetings,
> > 
> > * Tomas Vondra (tomas.vondra@2ndquadrant.com) wrote:
> > > On Wed, Jul 10, 2019 at 04:11:21PM -0400, Alvaro Herrera wrote:
> > > >On 2019-Jul-10, Bruce Momjian wrote:
> > > >
> > > >>Uh, what if a transaction modifies page 0 and page 1 of the same table
> > > >>--- don't those pages have the same LSN.
> > > >
> > > >No, because WAL being a physical change log, each page gets its own
> > > >WAL record with its own LSN.
> > > >
> > > 
> > > What if you have wal_log_hints=off? AFAIK that won't change the page LSN.
> > 
> > Alvaro suggested elsewhere that we require checksums for these, which
> > would also force wal_log_hints to be on, and therefore the LSN would
> > change.
> > 
> 
> Oh, I see - yes, that would solve the hint bits issue. Not sure we want
> to combine the features like this, though, as it increases the costs of
> TDE. But maybe it's the best solution.

Uh, why can't we just force log_hint_bits for encrypted tables?  Why
would we need to use checksums as well?

Why is page-number not needed in the nonce?  Because it is duplicative
of the LSN?  Can we use just LSN?  Do we need pg_class.oid too?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Joe Conway <mail@joeconway.com> wrote:

> Please see my other reply (and
> https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> appendix C as pointed out by Ryan downthread).

Thanks.

> At least in my mind, I trust a published specification from the
> nation-state level over random blogs or wikipedia. If we can find some
> equivalent published standards that contradict NIST we should discuss
> it, but for my money I would prefer to stick with the NIST recommended
> method to produce the IVs.

I don't think this as a problem of trusting A over B. Those blogs try to
explain the attacks in detail, while the NIST standard is just a set of
recommendations that does not (try to) provide technical details of comparable
depth.

Although I prefer understanding things in detail, I think it's o.k. to say in
documentation that "we use ... cipher because it complies to ... standard".

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Thu, Jul 11, 2019 at 06:45:36AM -0600, Ryan Lambert wrote:
> > >>Uh, what if a transaction modifies page 0 and page 1 of the same table
> 
> > >>--- don't those pages have the same LSN.
> > >
> > >No, because WAL being a physical change log, each page gets its own
> > >WAL record with its own LSN.
> > >
> >
> > What if you have wal_log_hints=off? AFAIK that won't change the page LSN.
> 
> >
> 
> > Alvaro suggested elsewhere that we require checksums for these, which
> > would also force wal_log_hints to be on, and therefore the LSN would
> > change.
>  
> Yes, it sounds like the agreement was LSN is unique when wal_log_hints is on. 
> I don't know enough about the internals to know if pg_class.oid is also needed
> or not.

Well, so, as far as we know now, every change to a heap/index page
advances the LSN, except for hint bits, which we can force to advance
LSN via wal_log_hints.  We automatically enable wal_log_hints for
checksums, so we can easily enable it automatically for encrypted
clusters.

I assume the LSN used for 8k pages and the segment numbers (for WAL) do
not overlap in numbering, for our nonce.

I think we will eventually have to have this setup verified by security
experts but I think we need to work through what is possible using
existing Postgres facilities before we present a possible solution.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jul 9, 2019 at 10:43 AM Bruce Momjian <bruce@momjian.us> wrote:
> FYI, pg_upgrade already preserves the pg_class.oid, which is why I
> recommended it over pg_class.relfilenode:

I think it's strange that pg_upgrade does not preserve the
relfilenode.  I think it would probably make more sense if it did.

Anyway, leaving that aside, you have to be able to read pg_class to
know the OID of a table, and you can't do that in recovery before
reaching consistency. Yet, you still need to be able to modify disk
blocks at that point, to finish recovery. So I can't see how any
system that involves figuring out the nonce from the OID would ever
work.

If we end up with random nonces, we're going to have to store them
someplace - either in some unencrypted portion of the disk blocks
themselves, or in a separate fork, or someplace else. If it's OK for
them to predictable as long as they vary a lot, we could derive them
from DBOID + RELFILENODE + FORK + BLOCK, but not from DBOID + RELOID +
FORK + BLOCK, because of the aforementioned recovery problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Jul 11, 2019 at 03:47:50PM -0400, Robert Haas wrote:
> On Tue, Jul 9, 2019 at 10:43 AM Bruce Momjian <bruce@momjian.us> wrote:
> > FYI, pg_upgrade already preserves the pg_class.oid, which is why I
> > recommended it over pg_class.relfilenode:
> 
> I think it's strange that pg_upgrade does not preserve the
> relfilenode.  I think it would probably make more sense if it did.
> 
> Anyway, leaving that aside, you have to be able to read pg_class to
> know the OID of a table, and you can't do that in recovery before
> reaching consistency. Yet, you still need to be able to modify disk
> blocks at that point, to finish recovery. So I can't see how any
> system that involves figuring out the nonce from the OID would ever
> work.
> 
> If we end up with random nonces, we're going to have to store them
> someplace - either in some unencrypted portion of the disk blocks
> themselves, or in a separate fork, or someplace else. If it's OK for
> them to predictable as long as they vary a lot, we could derive them
> from DBOID + RELFILENODE + FORK + BLOCK, but not from DBOID + RELOID +
> FORK + BLOCK, because of the aforementioned recovery problem.

Later in this thread, we decided that the page LSN was the best option
as a nonce because it changes every time the page chages.  (We will
enable wal_log_hints.)  We will not encrypt the first 16 bytes of the
page so it can be used for the nonce. (AES block ciphers are use 16-byte
blocks.)

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote:
> On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote:
> > Please see my other reply (and
> > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> > appendix C as pointed out by Ryan downthread).
> > 
> > At least in my mind, I trust a published specification from the
> > nation-state level over random blogs or wikipedia. If we can find some
> > equivalent published standards that contradict NIST we should discuss
> > it, but for my money I would prefer to stick with the NIST recommended
> > method to produce the IVs.
> 
> So, we have had a flurry of activity on this thread in the past day, so
> let me summarize:

Seems we have an updated approach:

First, we need to store the symmetric encryption key in the data
directory, like we do for SSL certificates and private keys.  (Crash
recovery needs access to this key, so we can't easily store it in a
database table.)  We will pattern it after the GUC
ssl_passphrase_command.   We will need to decide on a format for the
symmetric encryption key in the file so we can check that the supplied
passphrase properly unlocks the key.

Our first implementation will encrypt the entire cluster.  We can later
consider encryption per table or tablespace.  It is unclear if
encrypting different parts of the system with different keys is useful
or feasible.  (This is separate from key rotation.)

We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
8k pages will use the LSN as a nonce, which will be encrypted to
generate the initialization vector (IV).  We will not encrypt the first
16 bytes of each pages so the LSN can be used in this way.  The WAL will
use the WAL file segment number as the nonce and the IV will be created
in the same way.

wal_log_hints will be enabled automatically in encryption mode, like we
do for checksum mode, so we never encrypt different 8k pages with the
same IV.

There will need to be a pg_control field to indicate that encryption is
in use.

Right now we don't support the online changing of a cluster's checksum
mode, so I suggest we create a utility like pg_checksums --enable to
allow offline key rotation.  Once we get online checksum mode changing
ability, we can look into use that for encryption key rotation.

Did I miss anything?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 7/11/19 6:37 PM, Bruce Momjian wrote:
> On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote:
>> On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote:
>>> Please see my other reply (and
>>> https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
>>> appendix C as pointed out by Ryan downthread).
>>>
>>> At least in my mind, I trust a published specification from the
>>> nation-state level over random blogs or wikipedia. If we can find some
>>> equivalent published standards that contradict NIST we should discuss
>>> it, but for my money I would prefer to stick with the NIST recommended
>>> method to produce the IVs.
>>
>> So, we have had a flurry of activity on this thread in the past day, so
>> let me summarize:
> 
> Seems we have an updated approach:

I tried to keep up with this thread, and may have failed, but comments
inline...

> First, we need to store the symmetric encryption key in the data
> directory, like we do for SSL certificates and private keys.  (Crash
> recovery needs access to this key, so we can't easily store it in a
> database table.)  We will pattern it after the GUC
> ssl_passphrase_command.   We will need to decide on a format for the
> symmetric encryption key in the file so we can check that the supplied
> passphrase properly unlocks the key.
> 
> Our first implementation will encrypt the entire cluster.  We can later
> consider encryption per table or tablespace.  It is unclear if
> encrypting different parts of the system with different keys is useful
> or feasible.  (This is separate from key rotation.)

I still object strongly to using a single key for the entire database. I
think we can use a single key for WAL, but we need some way to split the
heap so that multiple keys are used. If not by tablespace, then some
other method.

Regardless of the method to split the heap into different keys, I think
there should be an option for some tables to not be encrypted. If we
decide it must be all or nothing for the first implementation I guess I
could live with it but would be very disappointed.

The keys themselves should be in an file which is encrypted by a master
key. Obtaining the master key should be pattern it after the GUC
ssl_passphrase_command.

> We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> 8k pages will use the LSN as a nonce, which will be encrypted to
> generate the initialization vector (IV).  We will not encrypt the first
> 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> use the WAL file segment number as the nonce and the IV will be created
> in the same way.

I vote for AES 256 rather than 128.

Did we determine that we no longer need table oid because LSNs are
sufficiently unique?

> wal_log_hints will be enabled automatically in encryption mode, like we
> do for checksum mode, so we never encrypt different 8k pages with the
> same IV.

check

> There will need to be a pg_control field to indicate that encryption is
> in use.

I didn't see that discussed but it makes sense.

> Right now we don't support the online changing of a cluster's checksum
> mode, so I suggest we create a utility like pg_checksums --enable to
> allow offline key rotation.  Once we get online checksum mode changing
> ability, we can look into use that for encryption key rotation.

Master key rotation should be trivial if we do it the way I discussed
above. Rotating the individual heap and WAL keys would certainly be a
bigger problem.

Thinking out loud (and I believe somewhere in this massive thread
someone else already said this), if we had a way to flag "key version"
at the page level it seems like we could potentially rekey page-by-page
while online, locking only one page at a time. We really only need to
support 2 key versions and could ping-pong between them as they change.
Or maybe this is a crazy idea.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
> On 7/11/19 6:37 PM, Bruce Momjian wrote:
> > Our first implementation will encrypt the entire cluster.  We can later
> > consider encryption per table or tablespace.  It is unclear if
> > encrypting different parts of the system with different keys is useful
> > or feasible.  (This is separate from key rotation.)
> 
> I still object strongly to using a single key for the entire database. I
> think we can use a single key for WAL, but we need some way to split the
> heap so that multiple keys are used. If not by tablespace, then some
> other method.

What do you base this on?

> Regardless of the method to split the heap into different keys, I think
> there should be an option for some tables to not be encrypted. If we
> decide it must be all or nothing for the first implementation I guess I
> could live with it but would be very disappointed.

What does it mean you "could live this it"?  Why do you consider having
some tables unencrypted important?

> The keys themselves should be in an file which is encrypted by a master
> key. Obtaining the master key should be pattern it after the GUC
> ssl_passphrase_command.
> 
> > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> > 8k pages will use the LSN as a nonce, which will be encrypted to
> > generate the initialization vector (IV).  We will not encrypt the first
> > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> > use the WAL file segment number as the nonce and the IV will be created
> > in the same way.
> 
> I vote for AES 256 rather than 128.

Why?  This page seems to think 128 is sufficient:


https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc

    For practical purposes, 128-bit keys are sufficient to ensure security.
    The larger key sizes exist mostly to satisfy some US military
    regulations which call for the existence of several distinct "security
    levels", regardless of whether breaking the lowest level is already far
    beyond existing technology.

We might need to run some benchmarks to determine the overhead of going
to AES256, because I am unclear of the security value.

> Did we determine that we no longer need table oid because LSNs are
> sufficiently unique?

I think so.

> > wal_log_hints will be enabled automatically in encryption mode, like we
> > do for checksum mode, so we never encrypt different 8k pages with the
> > same IV.
> 
> check
> 
> > There will need to be a pg_control field to indicate that encryption is
> > in use.
> 
> I didn't see that discussed but it makes sense.

Yes, it seems to make sense, but was not discussed.

> > Right now we don't support the online changing of a cluster's checksum
> > mode, so I suggest we create a utility like pg_checksums --enable to
> > allow offline key rotation.  Once we get online checksum mode changing
> > ability, we can look into use that for encryption key rotation.
> 
> Master key rotation should be trivial if we do it the way I discussed
> above. Rotating the individual heap and WAL keys would certainly be a
> bigger problem.

Yes, sorry, master key rotation is simple.  It is encryption key
rotation that I think needs a tool.

> Thinking out loud (and I believe somewhere in this massive thread
> someone else already said this), if we had a way to flag "key version"
> at the page level it seems like we could potentially rekey page-by-page
> while online, locking only one page at a time. We really only need to
> support 2 key versions and could ping-pong between them as they change.
> Or maybe this is a crazy idea.

Yes, we did talk about this.  It is certainly possible, but we would
still need a tool to guarantee all pages are using the new version, so I
am not sure what per-page buys us except making the later check faster. 
I don't see this as a version-1 feature, frankly.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 11, 2019 at 06:37:41PM -0400, Bruce Momjian wrote:
> wal_log_hints will be enabled automatically in encryption mode, like we
> do for checksum mode, so we never encrypt different 8k pages with the
> same IV.

Checksum mode can be enabled in encrypted clusters to detect modified
data, since the checksum is encrypted.  The WAL already has checksums.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 12, 2019 at 7:37 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote:
> > On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote:
> > > Please see my other reply (and
> > > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> > > appendix C as pointed out by Ryan downthread).
> > >
> > > At least in my mind, I trust a published specification from the
> > > nation-state level over random blogs or wikipedia. If we can find some
> > > equivalent published standards that contradict NIST we should discuss
> > > it, but for my money I would prefer to stick with the NIST recommended
> > > method to produce the IVs.
> >
> > So, we have had a flurry of activity on this thread in the past day, so
> > let me summarize:
>
> Seems we have an updated approach:
>
> First, we need to store the symmetric encryption key in the data
> directory, like we do for SSL certificates and private keys.  (Crash
> recovery needs access to this key, so we can't easily store it in a
> database table.)  We will pattern it after the GUC
> ssl_passphrase_command.   We will need to decide on a format for the
> symmetric encryption key in the file so we can check that the supplied
> passphrase properly unlocks the key.
>
> Our first implementation will encrypt the entire cluster.  We can later
> consider encryption per table or tablespace.  It is unclear if
> encrypting different parts of the system with different keys is useful
> or feasible.  (This is separate from key rotation.)
>
> We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> 8k pages will use the LSN as a nonce, which will be encrypted to
> generate the initialization vector (IV).  We will not encrypt the first
> 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> use the WAL file segment number as the nonce and the IV will be created
> in the same way.
>
> wal_log_hints will be enabled automatically in encryption mode, like we
> do for checksum mode, so we never encrypt different 8k pages with the
> same IV.

I guess that different two pages can have the same LSN when a heap
update modifies both a page for old tuple and another page for new
tuple.

heapam.c:3707
        recptr = log_heap_update(relation, buffer,
                                 newbuf, &oldtup, heaptup,
                                 old_key_tuple,
                                 all_visible_cleared,
                                 all_visible_cleared_new);
        if (newbuf != buffer)
        {
            PageSetLSN(BufferGetPage(newbuf), recptr);
        }
        PageSetLSN(BufferGetPage(buffer), recptr);

Wouldn't it a problem?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Thank you for summarizing the discussion, it's really helpful. I'll
update the wiki page based on the summary.

On Fri, Jul 12, 2019 at 10:05 AM Bruce Momjian <bruce@momjian.us> wrote:
> > The keys themselves should be in an file which is encrypted by a master
> > key. Obtaining the master key should be pattern it after the GUC
> > ssl_passphrase_command.

+1.
I will update the patch set based on the decision on this thread.

> >
> > > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> > > 8k pages will use the LSN as a nonce, which will be encrypted to
> > > generate the initialization vector (IV).  We will not encrypt the first
> > > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> > > use the WAL file segment number as the nonce and the IV will be created
> > > in the same way.
> >
> > I vote for AES 256 rather than 128.
>
> Why?  This page seems to think 128 is sufficient:
>
>
https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
>
>         For practical purposes, 128-bit keys are sufficient to ensure security.
>         The larger key sizes exist mostly to satisfy some US military
>         regulations which call for the existence of several distinct "security
>         levels", regardless of whether breaking the lowest level is already far
>         beyond existing technology.
>
> We might need to run some benchmarks to determine the overhead of going
> to AES256, because I am unclear of the security value.

'openssl speed' will help to see the performance differences easily.
FWIW I got the following result in my environment (Intel(R) Core(TM)
i7-3770 CPU @ 3.40GHz).

$ openssl speed -evp aes-128-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     642449.60k   656404.63k   700231.23k   706461.71k   706051.44k

$ openssl speed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     466787.73k   496237.08k   503477.16k   507113.32k   508453.80k

Regarding the security value, I found an interesting post by Bruce Schneier.

https://www.schneier.com/blog/archives/2009/07/another_new_aes.html

"And for new applications I suggest that people don't use AES-256.
AES-128 provides more than enough security margin for the forseeable
future. But if you're already using AES-256, there's no reason to
change."

>
> > Did we determine that we no longer need table oid because LSNs are
> > sufficiently unique?
>
> I think so.
>
> > > wal_log_hints will be enabled automatically in encryption mode, like we
> > > do for checksum mode, so we never encrypt different 8k pages with the
> > > same IV.
> >
> > check
> >
> > > There will need to be a pg_control field to indicate that encryption is
> > > in use.
> >
> > I didn't see that discussed but it makes sense.
>
> Yes, it seems to make sense, but was not discussed.
>
> > > Right now we don't support the online changing of a cluster's checksum
> > > mode, so I suggest we create a utility like pg_checksums --enable to
> > > allow offline key rotation.  Once we get online checksum mode changing
> > > ability, we can look into use that for encryption key rotation.
> >
> > Master key rotation should be trivial if we do it the way I discussed
> > above. Rotating the individual heap and WAL keys would certainly be a
> > bigger problem.
>
> Yes, sorry, master key rotation is simple.  It is encryption key
> rotation that I think needs a tool.

Agreed.

To rotate the master key we can have a SQL function or dedicated SQL
command passing the new master key or the passphrase to postgres.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Jul 11, 2019 at 9:05 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
> > I vote for AES 256 rather than 128.
>
> Why?  This page seems to think 128 is sufficient:
>
>
https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
>
>         For practical purposes, 128-bit keys are sufficient to ensure security.
>         The larger key sizes exist mostly to satisfy some US military
>         regulations which call for the existence of several distinct "security
>         levels", regardless of whether breaking the lowest level is already far
>         beyond existing technology.
>
> We might need to run some benchmarks to determine the overhead of going
> to AES256, because I am unclear of the security value.

If the algorithm and key size is not going to be configurable then
better to lean toward the larger size, especially given the desire for
future proofing against standards evolution and potential for the
encrypted data to be very long lived. NIST recommends AES-128 or
higher but there are other publications that recommend AES-256 for
long term usage:

NIST - 2019 : Recommends AES-128, AES-192, or AES-256.
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf

NSA - 2016 : Recommends AES-256 for future quantum resistance.

https://apps.nsa.gov/iaarchive/library/ia-guidance/ia-solutions-for-classified/algorithm-guidance/cnsa-suite-and-quantum-computing-faq.cfm

ECRYPT - 2015 - Recommends AES-256 for future quantum resistance.
https://www.ecrypt.eu.org/csa/documents/PQC-whitepaper.pdf

ECRYPT - 2018 - Recommends AES-256 for long term use.
https://www.ecrypt.eu.org/csa/documents/D5.4-FinalAlgKeySizeProt.pdf

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

On Fri, Jul 12, 2019 at 03:20:37PM +0900, Masahiko Sawada wrote:
> Thank you for summarizing the discussion, it's really helpful. I'll
> update the wiki page based on the summary.
> 
> On Fri, Jul 12, 2019 at 10:05 AM Bruce Momjian <bruce@momjian.us> wrote:
> > > The keys themselves should be in an file which is encrypted by a master
> > > key. Obtaining the master key should be pattern it after the GUC
> > > ssl_passphrase_command.
> 
> +1.
> I will update the patch set based on the decision on this thread.

Thanks.

> > > > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> > > > 8k pages will use the LSN as a nonce, which will be encrypted to
> > > > generate the initialization vector (IV).  We will not encrypt the first
> > > > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> > > > use the WAL file segment number as the nonce and the IV will be created
> > > > in the same way.
> > >
> > > I vote for AES 256 rather than 128.
> >
> > Why?  This page seems to think 128 is sufficient:
> >
> >
https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
> >
> >         For practical purposes, 128-bit keys are sufficient to ensure security.
> >         The larger key sizes exist mostly to satisfy some US military
> >         regulations which call for the existence of several distinct "security
> >         levels", regardless of whether breaking the lowest level is already far
> >         beyond existing technology.
> >
> > We might need to run some benchmarks to determine the overhead of going
> > to AES256, because I am unclear of the security value.
> 
> 'openssl speed' will help to see the performance differences easily.
> FWIW I got the following result in my environment (Intel(R) Core(TM)
> i7-3770 CPU @ 3.40GHz).
> 
> $ openssl speed -evp aes-128-cbc
> type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
> aes-128-cbc     642449.60k   656404.63k   700231.23k   706461.71k   706051.44k
> 
> $ openssl speed -evp aes-256-cbc
> type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
> aes-256-cbc     466787.73k   496237.08k   503477.16k   507113.32k   508453.80k

I saw similar numbers on my Intel(R) Xeon(R) CPU E5620  @ 2.40GHz with
AES optimization enabled on the CPUs:

    $ grep -i '\<aes\>' /proc/cpuinfo | wc -l
    16

    Doing aes-128-cbc for 3s on 8192 size blocks: 254000 aes-128-cbc's in 3.00s
    Doing aes-256-cbc for 3s on 8192 size blocks: 182496 aes-256-cbc's in 3.00s

which shows AES256 as 40% slower than AES128, which matches the 40%
mentioned here:


https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc

    The larger key sizes imply some CPU overhead (+20% for a 192-bit key,
    +40% for a 256-bit key: internally, the AES is a sequence of "rounds"
    and the AES standard says that there shall be 10, 12 or 14 rounds, for a
    128-bit, 192-bit or 256-bit key, respectively). So there is some
    rational reason not to use a larger than necessary key.

> Regarding the security value, I found an interesting post by Bruce Schneier.
> 
> https://www.schneier.com/blog/archives/2009/07/another_new_aes.html
> 
> "And for new applications I suggest that people don't use AES-256.
> AES-128 provides more than enough security margin for the forseeable
> future. But if you're already using AES-256, there's no reason to
> change."

Yes, that is what I have heard too.  I think the additional number of
people who use encryption because of its lower overhead will greatly
outweigh the benefit of using AES256 vs AES128.

> > Yes, sorry, master key rotation is simple.  It is encryption key
> > rotation that I think needs a tool.
> 
> Agreed.
> 
> To rotate the master key we can have a SQL function or dedicated SQL
> command passing the new master key or the passphrase to postgres.

Well, depending on how we store the encryption key, we will probably
change the master key via a command-line tool like pgchecksums.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 12, 2019 at 07:26:21AM -0400, Sehrope Sarkuni wrote:
> On Thu, Jul 11, 2019 at 9:05 PM Bruce Momjian <bruce@momjian.us> wrote:
> >
> > On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
> > > I vote for AES 256 rather than 128.
> >
> > Why?  This page seems to think 128 is sufficient:
> >
> >
https://crypto.stackexchange.com/questions/20/what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
> >
> >         For practical purposes, 128-bit keys are sufficient to ensure security.
> >         The larger key sizes exist mostly to satisfy some US military
> >         regulations which call for the existence of several distinct "security
> >         levels", regardless of whether breaking the lowest level is already far
> >         beyond existing technology.
> >
> > We might need to run some benchmarks to determine the overhead of going
> > to AES256, because I am unclear of the security value.
> 
> If the algorithm and key size is not going to be configurable then
> better to lean toward the larger size, especially given the desire for
> future proofing against standards evolution and potential for the
> encrypted data to be very long lived. NIST recommends AES-128 or
> higher but there are other publications that recommend AES-256 for
> long term usage:
> 
> NIST - 2019 : Recommends AES-128, AES-192, or AES-256.
> https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf
> 
> NSA - 2016 : Recommends AES-256 for future quantum resistance.
>
https://apps.nsa.gov/iaarchive/library/ia-guidance/ia-solutions-for-classified/algorithm-guidance/cnsa-suite-and-quantum-computing-faq.cfm
> 
> ECRYPT - 2015 - Recommends AES-256 for future quantum resistance.
> https://www.ecrypt.eu.org/csa/documents/PQC-whitepaper.pdf
> 
> ECRYPT - 2018 - Recommends AES-256 for long term use.
> https://www.ecrypt.eu.org/csa/documents/D5.4-FinalAlgKeySizeProt.pdf

Oh, interesting.  Let's see what performance tests with the database
show.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 12, 2019 at 02:15:02PM +0900, Masahiko Sawada wrote:
> > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> > 8k pages will use the LSN as a nonce, which will be encrypted to
> > generate the initialization vector (IV).  We will not encrypt the first
> > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> > use the WAL file segment number as the nonce and the IV will be created
> > in the same way.
> >
> > wal_log_hints will be enabled automatically in encryption mode, like we
> > do for checksum mode, so we never encrypt different 8k pages with the
> > same IV.
> 
> I guess that different two pages can have the same LSN when a heap
> update modifies both a page for old tuple and another page for new
> tuple.
> 
> heapam.c:3707
>         recptr = log_heap_update(relation, buffer,
>                                  newbuf, &oldtup, heaptup,
>                                  old_key_tuple,
>                                  all_visible_cleared,
>                                  all_visible_cleared_new);
>         if (newbuf != buffer)
>         {
>             PageSetLSN(BufferGetPage(newbuf), recptr);
>         }
>         PageSetLSN(BufferGetPage(buffer), recptr);
> 
> Wouldn't it a problem?

I had the same question.  If someone does:

    UPDATE tab SET col = col + 1

then each row change gets its own LSN.  You are asking if an update that
just expires one row and adds it to a new page gets the same LSN.  I
don't know.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 12, 2019 at 12:41:19PM -0600, Ryan Lambert wrote:
> >> I vote for AES 256 rather than 128.
> >
> > Why?  This page seems to think 128 is sufficient:
> >
> >         https://crypto.stackexchange.com/questions/20/
> what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
> >
> >         For practical purposes, 128-bit keys are sufficient to ensure
> security.
> >         The larger key sizes exist mostly to satisfy some US military
> >         regulations which call for the existence of several distinct
> "security
> >         levels", regardless of whether breaking the lowest level is already
> far
> >         beyond existing technology.
> 
> After researching AES key sizes a bit more my vote is (surprisingly?) for
> AES-128.  My reasoning is about security, I did not consider performance
> impacts in my decision.

Thank you for this exhaustive research.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 7/12/19 2:45 PM, Bruce Momjian wrote:
> On Fri, Jul 12, 2019 at 12:41:19PM -0600, Ryan Lambert wrote:
>> >> I vote for AES 256 rather than 128.
>> >
>> > Why?  This page seems to think 128 is sufficient:
>> >
>> >         https://crypto.stackexchange.com/questions/20/
>> what-are-the-practical-differences-between-256-bit-192-bit-and-128-bit-aes-enc
>> >
>> >         For practical purposes, 128-bit keys are sufficient to ensure
>> security.
>> >         The larger key sizes exist mostly to satisfy some US military
>> >         regulations which call for the existence of several distinct
>> "security
>> >         levels", regardless of whether breaking the lowest level is already
>> far
>> >         beyond existing technology.
>>
>> After researching AES key sizes a bit more my vote is (surprisingly?) for
>> AES-128.  My reasoning is about security, I did not consider performance
>> impacts in my decision.
>
> Thank you for this exhaustive research.

First of all, that is a mischaracterization of the issue. That paper
also states:

"we describe several key derivation attacks of practical complexity on
AES-256 when its number of rounds is reduced to approximately that of
AES-128. The best previously published  attacks  on  such  variants
were  far  from  practical,  requiring  4  related  keys  and  2^120
time  to break a 9-round version of AES-256 [9], and 64 related keys and
2^172time to break a 10-round version of AES-256 ([9], see also [2]). In
this paper we describe an attack on 9-round AES-256 which can findits
complete 256-bit key in 239time by using only the simplest type of
related keys (in which the chosenplaintexts are encrypted under two keys
whose XOR difference can be chosen in many different ways).Our best
attack on 10-round AES-256 requires only two keys and 245time, but it
uses a stronger type ofrelated subkey attack. These attacks can be
extended into a quasi-practical 270attack on 11-round AES,and into a
trivial 226attack on 8-round AES."

Notice 2 key things about this:
1. The attacks described are against a reduced number of rounds. AES256
   is 14 rounds, not 9 or 10.
2, They are "related key" attacks, which can be avoided by not using
   related keys, and we certainly should not be doing that.

Also, please read the links that Sehrope sent earlier if you have not
done so. In particular this one:

https://www.ecrypt.eu.org/csa/documents/PQC-whitepaper.pdf

which says:

"Post-quantum cryptography is an area of cryptography in which systems
are studied under the  security  assumption  that  the  attacker  has  a
 quantum  computer. This  attack  model  is interesting because Shor has
shown a quantum algorithm that breaks RSA, ECC, and finite field
discrete logarithms in polynomial time.  This means that in this model
all commonly used public-key systems are no longer secure.Symmetric
cryptography is also affected but significantly less.  For systems that
do not rely on mathematical structures the main effect is that an
algorithm due to Grover halves the security level, i.e., breaking
AES-128 takes 2^64 quantum operations while current attacks take  2^128
steps.   While  this  is  a  big  change,  it  can  be  managed  quite
easily  by  doubling the key sizes, e.g., by deploying AES-256.  The
operations needed in Grover’s algorithm are inherently  sequential
which  has  led  some  to  doubt  that  even  264quantum  operations
are feasible, but since the remedy of changing to larger key sizes is
very inexpensive it is generally recommended to do so."

In addition, that first paper was written in 2010, yet in 2016 NSA
published one of the other documents referenced by Sehrope:


https://apps.nsa.gov/iaarchive/customcf/openAttachment.cfm?FilePath=/iad/library/ia-guidance/ia-solutions-for-classified/algorithm-guidance/assets/public/upload/CNSA-Suite-and-Quantum-Computing-FAQ.pdf&WpKes=aF6woL7fQp3dJiyWTFKrYn3ZZShmLnzECSjJhf

Which states:

"If you have already implemented Suite B algorithms using the larger
(for TOP SECRET) key sizes, you should continue to use those algorithms
and key sizes through this upcoming transition period. In many products
changing to a larger key size can be done via a configuration
change.Implementations using only the algorithms previously approved for
SECRET and below in Suite B should not be used in NSS. In more precise
terms this means that NSS (National Security Systems) should no longer use
 •ECDH and ECDSA with NIST P-256
 •SHA-256
 •AES-128
 •RSA with 2048-bit keys
 •Diffie-Hellman with 2048-bit keys"


I stand by my position. At a minimum, we need a choice of AES128 and AES256.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/11/19 9:05 PM, Bruce Momjian wrote:
> On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
>> On 7/11/19 6:37 PM, Bruce Momjian wrote:
>> > Our first implementation will encrypt the entire cluster.  We can later
>> > consider encryption per table or tablespace.  It is unclear if
>> > encrypting different parts of the system with different keys is useful
>> > or feasible.  (This is separate from key rotation.)
>>
>> I still object strongly to using a single key for the entire database. I
>> think we can use a single key for WAL, but we need some way to split the
>> heap so that multiple keys are used. If not by tablespace, then some
>> other method.
>
> What do you base this on?

I have stated this more than once, and you and Stephen discussed it
earlier as well, but will find all the links back into the thread and
references and address in a separate email.

>> Regardless of the method to split the heap into different keys, I think
>> there should be an option for some tables to not be encrypted. If we
>> decide it must be all or nothing for the first implementation I guess I
>> could live with it but would be very disappointed.
>
> What does it mean you "could live this it"?  Why do you consider having
> some tables unencrypted important?


I think it is pretty obvious isn't it? You have talked about the
performance impact. Why would I want to encrypt, for example a lookup
table, if there is nothing in that table warranting encryption?

I think in many if not most applications the sensitive data is limited
to much less than all of the tables, and I'd rather not take the hit for
those tables.


>> The keys themselves should be in an file which is encrypted by a master
>> key. Obtaining the master key should be pattern it after the GUC
>> ssl_passphrase_command.
>>
>> > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
>> > 8k pages will use the LSN as a nonce, which will be encrypted to
>> > generate the initialization vector (IV).  We will not encrypt the first
>> > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
>> > use the WAL file segment number as the nonce and the IV will be created
>> > in the same way.
>>
>> I vote for AES 256 rather than 128.
>
> Why?  This page seems to think 128 is sufficient:


Addressed in the other email


>> Thinking out loud (and I believe somewhere in this massive thread
>> someone else already said this), if we had a way to flag "key version"
>> at the page level it seems like we could potentially rekey page-by-page
>> while online, locking only one page at a time. We really only need to
>> support 2 key versions and could ping-pong between them as they change.
>> Or maybe this is a crazy idea.
>
> Yes, we did talk about this.  It is certainly possible, but we would
> still need a tool to guarantee all pages are using the new version, so I
> am not sure what per-page buys us except making the later check faster.
> I don't see this as a version-1 feature, frankly.

If we allow for say, 2 versions of the key to exist at any given time,
and if we could store that key version information on each page, we
could change the key from old to new without locking the entire table at
once, just locking one page at a time. Or at least that was my thinking.

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/13/19 9:38 AM, Joe Conway wrote:
> On 7/11/19 9:05 PM, Bruce Momjian wrote:
>> On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
>>> On 7/11/19 6:37 PM, Bruce Momjian wrote:
>>> > Our first implementation will encrypt the entire cluster.  We can later
>>> > consider encryption per table or tablespace.  It is unclear if
>>> > encrypting different parts of the system with different keys is useful
>>> > or feasible.  (This is separate from key rotation.)
>>>
>>> I still object strongly to using a single key for the entire database. I
>>> think we can use a single key for WAL, but we need some way to split the
>>> heap so that multiple keys are used. If not by tablespace, then some
>>> other method.
>>
>> What do you base this on?

Ok, so here we go. See links below. I skimmed through the entire thread
and FWIW it was exhausting.

To some extent this degenerated into a general search for relevant
information:

---
[1] and [2] show that at least some file system encryption uses a
different key per file.
---
[2] also shows that file system encryption uses a KDF (key derivation
function) which we may want to use ourselves. The analogy would be
per-table derived key instead of per file derived key. Note that KDF is
a safe way to derive a key and it is not the same as a "related key"
which was mentioned on another email as an attack vector.
---
[2] also says provides additional support for AES 256. It also mentions
CBC versus XTS -- I came across this elsewhere and it bears discussion:

"Currently, the following pairs of encryption modes are supported:

    AES-256-XTS for contents and AES-256-CTS-CBC for filenames
    AES-128-CBC for contents and AES-128-CTS-CBC for filenames
    Adiantum for both contents and filenames

If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.

AES-128-CBC was added only for low-powered embedded devices with crypto
accelerators such as CAAM or CESA that do not support XTS."
---
[2] also states this, which again makes me think in terms of table being
the moral equivalent to a file:

"Unlike dm-crypt, fscrypt operates at the filesystem level rather than
at the block device level. This allows it to encrypt different files
with different keys and to have unencrypted files on the same
filesystem. This is useful for multi-user systems where each user’s
data-at-rest needs to be cryptographically isolated from the others.
However, except for filenames, fscrypt does not encrypt filesystem
metadata."
---
[3] suggests 68 GB per key and unique IV in GCM mode.
---
[4] specifies 68 GB per key and unique IV in CTR mode -- this applies
directly to our proposal to use CTR for WAL.
---
[5] has this to say which seems independent of mode:

"When encrypting data with a symmetric block cipher, which uses blocks
of n bits, some security concerns begin to appear when the amount of
data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
blocks). This means a limit of more than 250 millions of terabytes,
which is sufficiently large not to be a problem. That's precisely why
AES was defined with 128-bit blocks, instead of the more common (at that
time) 64-bit blocks: so that data size is practically unlimited."

But goes on to say:
"I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
reach that number of bits the probability of a collision will grow
quickly and you will be way over 50% probability of a collision by the
time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
collision negligible I recommend encrypting no more than n*2^(n/4) bits
with the same key. In the case of AES that works out to 64GB"

It is hard to say if that recommendation is per key or per key+IV.
---
[6] shows that Azure SQL Database uses AES 256 for TDE. It also seems to
imply a single key is used although at one point it says "transparent
data encryption master key, also known as the transparent data
encryption protector". The term "master key" indicates that they likely
use derived keys under the covers.
---
[7] is generally useful read about how many of the things we have been
discussing are done in SQL Server
---
[8] was referenced by Sehrope. In addition to support for AES 256 for
long term use, table 5.1 is interesting. It lists CBC mode as "legacy"
but not "future".
---
[9] IETF RFC for KDF
---
[10] IETF RFC for Key wrapping -- this is probably how we should wrap
the master key with the Key Encryption Key (KEK) -- i.e. the outer key
provided by the user or command on postmaster start
---

Based on all of that I cannot find a requirement that we use more than
one key per database.

But I did find that files in an encrypted file system are encrypted with
derived keys from a master key, and I view this as analogous to what we
are doing.

As an aside to the specific question, I also found more evidence that
AES 256 is appropriate.

Joe

============================
[1]
https://www.postgresql.org/message-id/832657a1-b27a-a33a-5bc2-ce420f95f4be%40joeconway.com

[2]
https://www.postgresql.org/message-id/20190708194733.cztnwhqge4acepzw%40development

[3]
https://www.postgresql.org/message-id/20190708211811.sio5o36zxhps7snx%40momjian.us

[4] https://www.rfc-editor.org/pdfrfc/rfc3686.txt.pdf

[5]
https://security.stackexchange.com/questions/33434/rsa-maximum-bytes-to-encrypt-comparison-to-aes-in-terms-of-security

[6]
https://docs.microsoft.com/en-us/azure/sql-database/transparent-data-encryption-azure-sql?view=sql-server-2017

[7]

https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/transparent-data-encryption?view=sql-server-2017

[8] https://www.ecrypt.eu.org/csa/documents/D5.4-FinalAlgKeySizeProt.pdf

[9] https://www.ietf.org/rfc/rfc5869.txt

[10] https://tools.ietf.org/html/rfc3394

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote:
>On 7/13/19 9:38 AM, Joe Conway wrote:
>> On 7/11/19 9:05 PM, Bruce Momjian wrote:
>>> On Thu, Jul 11, 2019 at 08:41:52PM -0400, Joe Conway wrote:
>>>> On 7/11/19 6:37 PM, Bruce Momjian wrote:
>>>> > Our first implementation will encrypt the entire cluster.  We can later
>>>> > consider encryption per table or tablespace.  It is unclear if
>>>> > encrypting different parts of the system with different keys is useful
>>>> > or feasible.  (This is separate from key rotation.)
>>>>
>>>> I still object strongly to using a single key for the entire database. I
>>>> think we can use a single key for WAL, but we need some way to split the
>>>> heap so that multiple keys are used. If not by tablespace, then some
>>>> other method.
>>>
>>> What do you base this on?
>
>Ok, so here we go. See links below. I skimmed through the entire thread
>and FWIW it was exhausting.
>
>To some extent this degenerated into a general search for relevant
>information:
>
>---
>[1] and [2] show that at least some file system encryption uses a
>different key per file.
>---
>[2] also shows that file system encryption uses a KDF (key derivation
>function) which we may want to use ourselves. The analogy would be
>per-table derived key instead of per file derived key. Note that KDF is
>a safe way to derive a key and it is not the same as a "related key"
>which was mentioned on another email as an attack vector.
>---
>[2] also says provides additional support for AES 256. It also mentions
>CBC versus XTS -- I came across this elsewhere and it bears discussion:
>
>"Currently, the following pairs of encryption modes are supported:
>
>    AES-256-XTS for contents and AES-256-CTS-CBC for filenames
>    AES-128-CBC for contents and AES-128-CTS-CBC for filenames
>    Adiantum for both contents and filenames
>
>If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
>
>AES-128-CBC was added only for low-powered embedded devices with crypto
>accelerators such as CAAM or CESA that do not support XTS."
>---
>[2] also states this, which again makes me think in terms of table being
>the moral equivalent to a file:
>
>"Unlike dm-crypt, fscrypt operates at the filesystem level rather than
>at the block device level. This allows it to encrypt different files
>with different keys and to have unencrypted files on the same
>filesystem. This is useful for multi-user systems where each user’s
>data-at-rest needs to be cryptographically isolated from the others.
>However, except for filenames, fscrypt does not encrypt filesystem
>metadata."
>---
>[3] suggests 68 GB per key and unique IV in GCM mode.
>---
>[4] specifies 68 GB per key and unique IV in CTR mode -- this applies
>directly to our proposal to use CTR for WAL.
>---
>[5] has this to say which seems independent of mode:
>
>"When encrypting data with a symmetric block cipher, which uses blocks
>of n bits, some security concerns begin to appear when the amount of
>data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
>bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
>blocks). This means a limit of more than 250 millions of terabytes,
>which is sufficiently large not to be a problem. That's precisely why
>AES was defined with 128-bit blocks, instead of the more common (at that
>time) 64-bit blocks: so that data size is practically unlimited."
>

FWIW I was a bit confused at first, because the copy paste mangled the
formulas a bit - it should have been 2^(n/2) and n*2^(n/2).

>But goes on to say:
>"I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
>reach that number of bits the probability of a collision will grow
>quickly and you will be way over 50% probability of a collision by the
>time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
>collision negligible I recommend encrypting no more than n*2^(n/4) bits
>with the same key. In the case of AES that works out to 64GB"
>
>It is hard to say if that recommendation is per key or per key+IV.

Hmm, yeah. The question is what collisions they have in mind? Presumably
it's AES(block1,key) = AES(block2,key) in which case it'd be with fixed
IV, so per key+IV.

>---
>[6] shows that Azure SQL Database uses AES 256 for TDE. It also seems to
>imply a single key is used although at one point it says "transparent
>data encryption master key, also known as the transparent data
>encryption protector". The term "master key" indicates that they likely
>use derived keys under the covers.
>---
>[7] is generally useful read about how many of the things we have been
>discussing are done in SQL Server
>---
>[8] was referenced by Sehrope. In addition to support for AES 256 for
>long term use, table 5.1 is interesting. It lists CBC mode as "legacy"
>but not "future".
>---
>[9] IETF RFC for KDF
>---
>[10] IETF RFC for Key wrapping -- this is probably how we should wrap
>the master key with the Key Encryption Key (KEK) -- i.e. the outer key
>provided by the user or command on postmaster start
>---
>
>Based on all of that I cannot find a requirement that we use more than
>one key per database.
>
>But I did find that files in an encrypted file system are encrypted with
>derived keys from a master key, and I view this as analogous to what we
>are doing.
>

My understanding always was that we'd do something like that, i.e. we'd
have a master key (or perhaps multiple of them, for various users), but
the data would be encrypted with secondary (generated) keys, and those
secondary keys would be encrypted by the master key. At least that's
what was proposed at the beginning of this thread by Insung Moon.

But AFAICS the 2-tier key scheme is primarily motivated by operational
reasons, i.e. effort to rotate the master key etc. So I would not expect
to find recommendations to use multiple keys in sources primarily
dealing with cryptography.

One extra thing we should consider is authenticated encryption. We can't
just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
as that does not provide integrity protection (i.e. can't detect when
the ciphertext was corrupted due to disk failure or intentionally). And
we can't quite rely on checksums, because that checksums the plaintext
and is stored encrypted.

Which seems pretty annoying, because then the checksums won't verify
data as sent to the storage system, and verify checksums would require
access to all keys (how do you do that in offline mode?).

But the main issue with checksum-then-encrypt is it's essentially
"MAC-then-Encrypt" and that does not provide Authenticated Encryption
security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
in which case we'll need to store the MAC somewhere (probably in the
same place as the nonce/IV/key/... for each page).

I've also stumbled upon [2], which is a nice doctoral thesis about disk
encryption - in particular chapter 4 is a nice overview of the threat
model and use cases. That guy also had a nice talk at FOSDEM 2018 about
data dm-integrity etc. [3]

[1] https://www.cosic.esat.kuleuven.be/school-iot/slides/AuthenticatedEncryptionII.pdf

[2] https://is.muni.cz/th/vesfr/final.pdf

[3] https://ftp.fau.de/fosdem/2018/Janson/cryptsetup.mp4


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On 7/13/19 2:41 PM, Joe Conway wrote:
> [2]
> https://www.postgresql.org/message-id/20190708194733.cztnwhqge4acepzw%40development

BTW I managed to mess up this link. This is what I intended to link
there (from Tomas):

[2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html

I am sure I have confused the heck out of everyone reading what I wrote
by that error :-/

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 7/13/19 5:58 PM, Tomas Vondra wrote:
> On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote:
>>[2] also says provides additional support for AES 256. It also mentions
>>CBC versus XTS -- I came across this elsewhere and it bears discussion:
>>
>>"Currently, the following pairs of encryption modes are supported:
>>
>>    AES-256-XTS for contents and AES-256-CTS-CBC for filenames
>>    AES-128-CBC for contents and AES-128-CTS-CBC for filenames
>>    Adiantum for both contents and filenames
>>
>>If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
>>
>>AES-128-CBC was added only for low-powered embedded devices with crypto
>>accelerators such as CAAM or CESA that do not support XTS."
>>---
>>[2] also states this, which again makes me think in terms of table being
>>the moral equivalent to a file:
>>
>>"Unlike dm-crypt, fscrypt operates at the filesystem level rather than
>>at the block device level. This allows it to encrypt different files
>>with different keys and to have unencrypted files on the same
>>filesystem. This is useful for multi-user systems where each user’s
>>data-at-rest needs to be cryptographically isolated from the others.
>>However, except for filenames, fscrypt does not encrypt filesystem
>>metadata."

<snip>

>>[5] has this to say which seems independent of mode:
>>
>>"When encrypting data with a symmetric block cipher, which uses blocks
>>of n bits, some security concerns begin to appear when the amount of
>>data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
>>bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
>>blocks). This means a limit of more than 250 millions of terabytes,
>>which is sufficiently large not to be a problem. That's precisely why
>>AES was defined with 128-bit blocks, instead of the more common (at that
>>time) 64-bit blocks: so that data size is practically unlimited."
>>
>
> FWIW I was a bit confused at first, because the copy paste mangled the
> formulas a bit - it should have been 2^(n/2) and n*2^(n/2).

Yeah, sorry about that.

>>But goes on to say:
>>"I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
>>reach that number of bits the probability of a collision will grow
>>quickly and you will be way over 50% probability of a collision by the
>>time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
>>collision negligible I recommend encrypting no more than n*2^(n/4) bits
>>with the same key. In the case of AES that works out to 64GB"
>>
>>It is hard to say if that recommendation is per key or per key+IV.
>
> Hmm, yeah. The question is what collisions they have in mind? Presumably
> it's AES(block1,key) = AES(block2,key) in which case it'd be with fixed
> IV, so per key+IV.

Seems likely.

>>But I did find that files in an encrypted file system are encrypted with
>>derived keys from a master key, and I view this as analogous to what we
>>are doing.
>>
>
> My understanding always was that we'd do something like that, i.e. we'd
> have a master key (or perhaps multiple of them, for various users), but
> the data would be encrypted with secondary (generated) keys, and those
> secondary keys would be encrypted by the master key. At least that's
> what was proposed at the beginning of this thread by Insung Moon.

In my email I linked the wrong page for [2]. The correct one is here:
[2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html

Following that, I think we could end up with three tiers:

1. A master key encryption key (KEK): this is the ley supplied by the
   database admin using something akin to ssl_passphrase_command

2. A master data encryption key (MDEK): this is a generated key using a
   cryptographically secure pseudo-random number generator. It is
   encrypted using the KEK, probably with Key Wrap (KW):
   or maybe better Key Wrap with Padding (KWP):

3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
    table specific keys.

3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
    generate new keys when needed for WAL (based on the other info we
    need to change WAL keys every 68 GB unless I read that wrong).

I believe that would allows us to have multiple keys but they are
derived securely from the one DEK using available info similar to the
way we intend to use LSN to derive the IVs -- perhaps table.oid for
tables and something else for WAL.

We also need to figure out how/when to generate new WDEK. Maybe every
checkpoint, also meaning we would have to force a checkpoint every 68GB?

[HKDF]: https://tools.ietf.org/html/rfc5869
[KW]: https://tools.ietf.org/html/rfc3394
[KWP]: https://tools.ietf.org/html/rfc5649


> But AFAICS the 2-tier key scheme is primarily motivated by operational
> reasons, i.e. effort to rotate the master key etc. So I would not expect
> to find recommendations to use multiple keys in sources primarily
> dealing with cryptography.

It does in [2]


> One extra thing we should consider is authenticated encryption. We can't
> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> as that does not provide integrity protection (i.e. can't detect when
> the ciphertext was corrupted due to disk failure or intentionally). And
> we can't quite rely on checksums, because that checksums the plaintext
> and is stored encrypted.

I agree that authenticated encryption would be a good goal. I'm not sure
we need to require it for the first version, although it would mean
another option for the encryption type. That may be another good reason
to allow both AES 128 and AES 256 CTR/CBC in the first version, as it
will hopefully ensure that when we add different modes later it will be
less painful.

We could check the CRC prior to encryption and throw an ERROR if it is
not correct. After decryption we can check it again -- if it no longer
matches we would know there way a corruption or change of the
ciphertext, no?

Hmm, I guess the entire page of ciphertext could be faked including CRC,
so this would only really cover corruption, not an intentional change if
it were done properly.

> Which seems pretty annoying, because then the checksums won't verify
> data as sent to the storage system, and verify checksums would require
> access to all keys (how do you do that in offline mode?).

Given the scheme above I don't see why that would be an issue. The keys
are all accessible via the MDEK, which is in turn available via the KEK.

> But the main issue with checksum-then-encrypt is it's essentially
> "MAC-then-Encrypt" and that does not provide Authenticated Encryption
> security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
> in which case we'll need to store the MAC somewhere (probably in the
> same place as the nonce/IV/key/... for each page).


Yeah, that's why I think maybe this is a v2 feature.


> I've also stumbled upon [2], which is a nice doctoral thesis about disk
> encryption - in particular chapter 4 is a nice overview of the threat
> model and use cases. That guy also had a nice talk at FOSDEM 2018 about
> data dm-integrity etc. [3]
>
> [1] https://www.cosic.esat.kuleuven.be/school-iot/slides/AuthenticatedEncryptionII.pdf
> [2] https://is.muni.cz/th/vesfr/final.pdf
> [3] https://ftp.fau.de/fosdem/2018/Janson/cryptsetup.mp4

Awesome links -- thanks!

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote:
>On 7/13/19 5:58 PM, Tomas Vondra wrote:
>> On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote:
>>>[2] also says provides additional support for AES 256. It also mentions
>>>CBC versus XTS -- I came across this elsewhere and it bears discussion:
>>>
>>>"Currently, the following pairs of encryption modes are supported:
>>>
>>>    AES-256-XTS for contents and AES-256-CTS-CBC for filenames
>>>    AES-128-CBC for contents and AES-128-CTS-CBC for filenames
>>>    Adiantum for both contents and filenames
>>>
>>>If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
>>>
>>>AES-128-CBC was added only for low-powered embedded devices with crypto
>>>accelerators such as CAAM or CESA that do not support XTS."
>>>---
>>>[2] also states this, which again makes me think in terms of table being
>>>the moral equivalent to a file:
>>>
>>>"Unlike dm-crypt, fscrypt operates at the filesystem level rather than
>>>at the block device level. This allows it to encrypt different files
>>>with different keys and to have unencrypted files on the same
>>>filesystem. This is useful for multi-user systems where each user’s
>>>data-at-rest needs to be cryptographically isolated from the others.
>>>However, except for filenames, fscrypt does not encrypt filesystem
>>>metadata."
>
><snip>
>
>>>[5] has this to say which seems independent of mode:
>>>
>>>"When encrypting data with a symmetric block cipher, which uses blocks
>>>of n bits, some security concerns begin to appear when the amount of
>>>data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
>>>bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
>>>blocks). This means a limit of more than 250 millions of terabytes,
>>>which is sufficiently large not to be a problem. That's precisely why
>>>AES was defined with 128-bit blocks, instead of the more common (at that
>>>time) 64-bit blocks: so that data size is practically unlimited."
>>>
>>
>> FWIW I was a bit confused at first, because the copy paste mangled the
>> formulas a bit - it should have been 2^(n/2) and n*2^(n/2).
>
>Yeah, sorry about that.
>
>>>But goes on to say:
>>>"I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
>>>reach that number of bits the probability of a collision will grow
>>>quickly and you will be way over 50% probability of a collision by the
>>>time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
>>>collision negligible I recommend encrypting no more than n*2^(n/4) bits
>>>with the same key. In the case of AES that works out to 64GB"
>>>
>>>It is hard to say if that recommendation is per key or per key+IV.
>>
>> Hmm, yeah. The question is what collisions they have in mind? Presumably
>> it's AES(block1,key) = AES(block2,key) in which case it'd be with fixed
>> IV, so per key+IV.
>
>Seems likely.
>
>>>But I did find that files in an encrypted file system are encrypted with
>>>derived keys from a master key, and I view this as analogous to what we
>>>are doing.
>>>
>>
>> My understanding always was that we'd do something like that, i.e. we'd
>> have a master key (or perhaps multiple of them, for various users), but
>> the data would be encrypted with secondary (generated) keys, and those
>> secondary keys would be encrypted by the master key. At least that's
>> what was proposed at the beginning of this thread by Insung Moon.
>
>In my email I linked the wrong page for [2]. The correct one is here:
>[2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html
>
>Following that, I think we could end up with three tiers:
>
>1. A master key encryption key (KEK): this is the ley supplied by the
>   database admin using something akin to ssl_passphrase_command
>
>2. A master data encryption key (MDEK): this is a generated key using a
>   cryptographically secure pseudo-random number generator. It is
>   encrypted using the KEK, probably with Key Wrap (KW):
>   or maybe better Key Wrap with Padding (KWP):
>
>3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
>    table specific keys.
>
>3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
>    generate new keys when needed for WAL (based on the other info we
>    need to change WAL keys every 68 GB unless I read that wrong).
>
>I believe that would allows us to have multiple keys but they are
>derived securely from the one DEK using available info similar to the
>way we intend to use LSN to derive the IVs -- perhaps table.oid for
>tables and something else for WAL.
>
>We also need to figure out how/when to generate new WDEK. Maybe every
>checkpoint, also meaning we would have to force a checkpoint every 68GB?
>

I think that very much depends on what exactly the 68GB refers to - key
or key+IV? If key+IV, then I suppose we can use LSN as IV and we would
not need to change checkpoints. But it's not clear to me why we would
need to force checkpoints at all? Surely we can just write a WAL message
about switching to the new key, or something like that?

>[HKDF]: https://tools.ietf.org/html/rfc5869
>[KW]: https://tools.ietf.org/html/rfc3394
>[KWP]: https://tools.ietf.org/html/rfc5649
>
>
>> But AFAICS the 2-tier key scheme is primarily motivated by operational
>> reasons, i.e. effort to rotate the master key etc. So I would not expect
>> to find recommendations to use multiple keys in sources primarily
>> dealing with cryptography.
>
>It does in [2]
>
>
>> One extra thing we should consider is authenticated encryption. We can't
>> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> as that does not provide integrity protection (i.e. can't detect when
>> the ciphertext was corrupted due to disk failure or intentionally). And
>> we can't quite rely on checksums, because that checksums the plaintext
>> and is stored encrypted.
>
>I agree that authenticated encryption would be a good goal. I'm not sure
>we need to require it for the first version, although it would mean
>another option for the encryption type. That may be another good reason
>to allow both AES 128 and AES 256 CTR/CBC in the first version, as it
>will hopefully ensure that when we add different modes later it will be
>less painful.
>
>We could check the CRC prior to encryption and throw an ERROR if it is
>not correct. After decryption we can check it again -- if it no longer
>matches we would know there way a corruption or change of the
>ciphertext, no?
>
>Hmm, I guess the entire page of ciphertext could be faked including CRC,
>so this would only really cover corruption, not an intentional change if
>it were done properly.
>

I don't think any of the schemes discussed here provides protection
against this sort of replay attacks (i.e. replacing a page with an older
copy of the page). That would probably require having some global
checksum or something like that.

>> Which seems pretty annoying, because then the checksums won't verify
>> data as sent to the storage system, and verify checksums would require
>> access to all keys (how do you do that in offline mode?).
>
>Given the scheme above I don't see why that would be an issue. The keys
>are all accessible via the MDEK, which is in turn available via the KEK.
>

I just don't know how the offline tools will access the KMS to get the
keys. But maybe that's not an issue. But even then I think it's kinda
against the idea of checksums that they would not checksum what was sent
to the storage system.

>> But the main issue with checksum-then-encrypt is it's essentially
>> "MAC-then-Encrypt" and that does not provide Authenticated Encryption
>> security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
>> in which case we'll need to store the MAC somewhere (probably in the
>> same place as the nonce/IV/key/... for each page).
>
>
>Yeah, that's why I think maybe this is a v2 feature.
>

Maybe - as long as we design it with enough flexibility to enable it
later, that might work. That depends on where we store the metadata,
etc.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

> On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote:
> >On 7/13/19 9:38 AM, Joe Conway wrote:

> >[5] has this to say which seems independent of mode:
> >
> >"When encrypting data with a symmetric block cipher, which uses blocks
> >of n bits, some security concerns begin to appear when the amount of
> >data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
> >bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
> >blocks). This means a limit of more than 250 millions of terabytes,
> >which is sufficiently large not to be a problem. That's precisely why
> >AES was defined with 128-bit blocks, instead of the more common (at that
> >time) 64-bit blocks: so that data size is practically unlimited."
> >
> 
> FWIW I was a bit confused at first, because the copy paste mangled the
> formulas a bit - it should have been 2^(n/2) and n*2^(n/2).
> 
> >But goes on to say:
> >"I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
> >reach that number of bits the probability of a collision will grow
> >quickly and you will be way over 50% probability of a collision by the
> >time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
> >collision negligible I recommend encrypting no more than n*2^(n/4) bits
> >with the same key. In the case of AES that works out to 64GB"
> >
> >It is hard to say if that recommendation is per key or per key+IV.
> 
> Hmm, yeah. The question is what collisions they have in mind? Presumably
> it's AES(block1,key) = AES(block2,key) in which case it'd be with fixed
> IV, so per key+IV.

I've spent a while trying to understand where the formula comes from. If the
problem can be expressed as "avoidance of repeating blocks of ciphertext",
then it's basically the known "birthday problem". Then we can use this formula
[1]

n ~ sqrt(2 * m * p(n))

(note that the meaning of "n" is different form the formula introduced
upthread) and substitute

1) 0.5 for the probability p(n)

2) 2^b for the number of distinct blocks "m", where "b" is number of bits in
an encryption block

Then the formula becomes

    n ~ sqrt(2^b)

and thus

    n ~ 2^(b/2)

So if the number of safely encrypted blocks was derived this way, I agree that
IV was not taken into consideration: if there is an IV, then identical blocks
of ciphertext are not a problem because they represent different blocks of
plaintext.


[1] https://en.wikipedia.org/wiki/Birthday_problem#Square_approximation

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> On Mon, Jun 17, 2019 at 11:02 PM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> > On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote:
> > >On 6/17/19 8:29 AM, Masahiko Sawada wrote:
> > >> From perspective of  cryptographic, I think the fine grained TDE would
> > >> be better solution. Therefore if we eventually want the fine grained
> > >> TDE I wonder if it might be better to develop the table/tablespace TDE
> > >> first while keeping it simple as much as possible in v1, and then we
> > >> can provide the functionality to encrypt other data in database
> > >> cluster to satisfy the encrypting-everything requirement. I guess that
> > >> it's easier to incrementally add encryption target objects rather than
> > >> making it fine grained while not changing encryption target objects.
> > >>
> > >> FWIW I'm writing a draft patch of per tablespace TDE and will submit
> > >> it in this month. We can more discuss the complexity of the proposed
> > >> TDE using it.
> > >
> > >+1
> > >
> > >Looking forward to it.
> > >
> >
> > Yep. In particular, I'm interested in those aspects:
> >
> 
> Attached the draft version patch sets of per tablespace transparent
> data at rest encryption.

I was worried that there's competition between us but now that I've checked
your patch set I see that you already use some parts of

https://commitfest.postgresql.org/23/2104/

although not the latest version. I'm supposed to work on the encryption now,
so thinking what to do next. I think we should coordinate the effort, possibly
off-list. The earlier we have a single patch set the more efficient the work
should be.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Sat, Jul 13, 2019 at 09:38:06AM -0400, Joe Conway wrote:
> On 7/11/19 9:05 PM, Bruce Momjian wrote:
> >> Regardless of the method to split the heap into different keys, I think
> >> there should be an option for some tables to not be encrypted. If we
> >> decide it must be all or nothing for the first implementation I guess I
> >> could live with it but would be very disappointed.
> > 
> > What does it mean you "could live this it"?  Why do you consider having
> > some tables unencrypted important?
> 
> 
> I think it is pretty obvious isn't it? You have talked about the
> performance impact. Why would I want to encrypt, for example a lookup
> table, if there is nothing in that table warranting encryption?

Well, a lookup table is not going to get many writes, and will usually
stay in shared buffers, so I don't see the value in skipping encryption
of it.  However, the big issue is that having encryption on only some
tables has a cost, both in code complexity, and in looking up pg_class
or pg_tablespace rows to find out what needs encryption.  Also, it would
lead to information leakage if we don't encrypt all of WAL but instead
only encrypt the entries for tables/tablespaces that need encryption. 
(I don't think we discussed how WAL would be encrypted in such cases.)

My point is that doing encryption of only some data might actually make
the system slower due to the lookups, so I think we need to implement
all-cluster encryption and then see what the overhead is, and if there
are use-cases for not encrypting only some data.

> I think in many if not most applications the sensitive data is limited
> to much less than all of the tables, and I'd rather not take the hit for
> those tables.

Agreed, but let's see what the overhead it first, and also decide how
WAL would be handled in such cases.

> >> Thinking out loud (and I believe somewhere in this massive thread
> >> someone else already said this), if we had a way to flag "key version"
> >> at the page level it seems like we could potentially rekey page-by-page
> >> while online, locking only one page at a time. We really only need to
> >> support 2 key versions and could ping-pong between them as they change.
> >> Or maybe this is a crazy idea.
> > 
> > Yes, we did talk about this.  It is certainly possible, but we would
> > still need a tool to guarantee all pages are using the new version, so I
> > am not sure what per-page buys us except making the later check faster. 
> > I don't see this as a version-1 feature, frankly.
> 
> If we allow for say, 2 versions of the key to exist at any given time,
> and if we could store that key version information on each page, we
> could change the key from old to new without locking the entire table at
> once, just locking one page at a time. Or at least that was my thinking.

Yes, that is true, but eventually we will need to do key rotation again,
so we have to be sure the old key is no longer being used, so we would
still need a tool to check everything to be sure all the pages are using
the new key, and that could be done a page at a time.

It does feel like our checksum problem and I am hoping the
infrastructure to allow that to be done online can be used for this too.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 13, 2019 at 02:41:34PM -0400, Joe Conway wrote:
> On 7/13/19 9:38 AM, Joe Conway wrote:
> ---
> [1] and [2] show that at least some file system encryption uses a
> different key per file.

Yes, I see later they did that for per-file keys, but I think with WAL
and crash recovery we decided there was little value in trying that
since all keys would need to be online for recovery.

> ---
> [2] also shows that file system encryption uses a KDF (key derivation
> function) which we may want to use ourselves. The analogy would be
> per-table derived key instead of per file derived key. Note that KDF is
> a safe way to derive a key and it is not the same as a "related key"
> which was mentioned on another email as an attack vector.
> ---
> [2] also says provides additional support for AES 256. It also mentions
> CBC versus XTS -- I came across this elsewhere and it bears discussion:
> 
> "Currently, the following pairs of encryption modes are supported:
> 
>     AES-256-XTS for contents and AES-256-CTS-CBC for filenames
>     AES-128-CBC for contents and AES-128-CTS-CBC for filenames
>     Adiantum for both contents and filenames
> 
> If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
> 
> AES-128-CBC was added only for low-powered embedded devices with crypto
> accelerators such as CAAM or CESA that do not support XTS."

It would be nice to understand what XTS adds to CBC.

> [5] has this to say which seems independent of mode:
> 
> "When encrypting data with a symmetric block cipher, which uses blocks
> of n bits, some security concerns begin to appear when the amount of
> data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2
> bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit
> blocks). This means a limit of more than 250 millions of terabytes,
> which is sufficiently large not to be a problem. That's precisely why
> AES was defined with 128-bit blocks, instead of the more common (at that
> time) 64-bit blocks: so that data size is practically unlimited."
> 
> But goes on to say:
> "I wouldn't use n*2^(n/2) bits in any sort of recommendation. Once you
> reach that number of bits the probability of a collision will grow
> quickly and you will be way over 50% probability of a collision by the
> time you reach 2*n*2^(n/2) bits. In order to keep the probability of a
> collision negligible I recommend encrypting no more than n*2^(n/4) bits
> with the same key. In the case of AES that works out to 64GB"
>
> It is hard to say if that recommendation is per key or per key+IV.

When they mention collision, are they assuming a random nonce?  I am
guessing they do, I think the LSN avoids that problem because we
effectively have a counter.

> ---
> [6] shows that Azure SQL Database uses AES 256 for TDE. It also seems to
> imply a single key is used although at one point it says "transparent
> data encryption master key, also known as the transparent data
> encryption protector". The term "master key" indicates that they likely
> use derived keys under the covers.
> ---
> [7] is generally useful read about how many of the things we have been
> discussing are done in SQL Server
> ---
> [8] was referenced by Sehrope. In addition to support for AES 256 for
> long term use, table 5.1 is interesting. It lists CBC mode as "legacy"
> but not "future".

Interesting.  Is the a reason stated?

> ---
> [9] IETF RFC for KDF
> ---
> [10] IETF RFC for Key wrapping -- this is probably how we should wrap
> the master key with the Key Encryption Key (KEK) -- i.e. the outer key
> provided by the user or command on postmaster start

Yes, I think we all agreed to have a passphrase to lock the encryption
keys.

> ---
> 
> Based on all of that I cannot find a requirement that we use more than
> one key per database.

You mean cluster, right?  That is great news!

> But I did find that files in an encrypted file system are encrypted with
> derived keys from a master key, and I view this as analogous to what we
> are doing.

Agreed.

> As an aside to the specific question, I also found more evidence that
> AES 256 is appropriate.

I think we should allow the AES128/AES256 to be optional on version 1 of
the feature, or at least call the initdb option --encrypt-aes128, like
we did with SCRAM, so we have a clear path to adding AES256.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
> One extra thing we should consider is authenticated encryption. We can't
> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> as that does not provide integrity protection (i.e. can't detect when
> the ciphertext was corrupted due to disk failure or intentionally). And
> we can't quite rely on checksums, because that checksums the plaintext
> and is stored encrypted.

Uh, if someone modifies a few bytes of the page, we will decrypt it, but
the checksum (per-page or WAL) will not match our decrypted output.  How
would they make it match the checksum without already knowing the key. 
I read [1] but could not see that explained.

This post discussed it:

    https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac

I realize in a new system we might prefer encrypt-then-mac, TLS and SSL
do it differently, and I don't think the security problems of
MAC-then-Encrypt apply to our use-case, e.g. API programming errors.

If we want to go crazy, we could encrypt, assume zeros for the CRC,
compute the MAC and put it in the place of the CRC is, but then tools
that read CRC would see that as an error, so we don't want to go there.
Yes, crazy.

> Which seems pretty annoying, because then the checksums won't verify
> data as sent to the storage system, and verify checksums would require
> access to all keys (how do you do that in offline mode?).

Uh, the keys are stored in a PGDATA file --- seems simple enough, but we
would either have to do whole-cluster encryption or have some per-page
encryption flag.

> But the main issue with checksum-then-encrypt is it's essentially
> "MAC-then-Encrypt" and that does not provide Authenticated Encryption
> security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
> in which case we'll need to store the MAC somewhere (probably in the
> same place as the nonce/IV/key/... for each page).

I don't think we are planning to store the nonce/IV on each page but
rather use the LSN (already on the page), and perhaps in addition, the
page number.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote:
> On 7/13/19 5:58 PM, Tomas Vondra wrote:
> In my email I linked the wrong page for [2]. The correct one is here:
> [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html
> 
> Following that, I think we could end up with three tiers:
> 
> 1. A master key encryption key (KEK): this is the ley supplied by the
>    database admin using something akin to ssl_passphrase_command
> 
> 2. A master data encryption key (MDEK): this is a generated key using a
>    cryptographically secure pseudo-random number generator. It is
>    encrypted using the KEK, probably with Key Wrap (KW):
>    or maybe better Key Wrap with Padding (KWP):
> 
> 3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
>     table specific keys.

Uh, when was per-table encryption keys discussed?  Uses pg_class.oid or
relfilenode?

> 3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
>     generate new keys when needed for WAL (based on the other info we
>     need to change WAL keys every 68 GB unless I read that wrong).

I thought we were going to use the WAL segement number for each 16MB
file so eachy 16MB gets a new nonce.

> I believe that would allows us to have multiple keys but they are
> derived securely from the one DEK using available info similar to the
> way we intend to use LSN to derive the IVs -- perhaps table.oid for
> tables and something else for WAL.

Ah, got it.  We might want to use relfilenode (and have pg_upgrade
preserve it) to avoid having to do catalog lookups during WAL recovery.
However, I thought we were still unclear if that 68GB is per secret or
per nonce/secret.

> > One extra thing we should consider is authenticated encryption. We can't
> > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> > as that does not provide integrity protection (i.e. can't detect when
> > the ciphertext was corrupted due to disk failure or intentionally). And
> > we can't quite rely on checksums, because that checksums the plaintext
> > and is stored encrypted.
> 
> I agree that authenticated encryption would be a good goal. I'm not sure
> we need to require it for the first version, although it would mean
> another option for the encryption type. That may be another good reason
> to allow both AES 128 and AES 256 CTR/CBC in the first version, as it
> will hopefully ensure that when we add different modes later it will be
> less painful.
> 
> We could check the CRC prior to encryption and throw an ERROR if it is
> not correct. After decryption we can check it again -- if it no longer
> matches we would know there way a corruption or change of the
> ciphertext, no?

Yes, that is my hope too.

> Hmm, I guess the entire page of ciphertext could be faked including CRC,
> so this would only really cover corruption, not an intentional change if
> it were done properly.

Uh, how would they get a CRC to decrypt to match their page contents?

> > Which seems pretty annoying, because then the checksums won't verify
> > data as sent to the storage system, and verify checksums would require
> > access to all keys (how do you do that in offline mode?).
> 
> Given the scheme above I don't see why that would be an issue. The keys
> are all accessible via the MDEK, which is in turn available via the KEK.

Yep.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul 15, 2019 at 03:47:59AM +0200, Tomas Vondra wrote:
> On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote:
> > We could check the CRC prior to encryption and throw an ERROR if it is
> > not correct. After decryption we can check it again -- if it no longer
> > matches we would know there way a corruption or change of the
> > ciphertext, no?
> > 
> > Hmm, I guess the entire page of ciphertext could be faked including CRC,
> > so this would only really cover corruption, not an intentional change if
> > it were done properly.
> > 
> 
> I don't think any of the schemes discussed here provides protection
> against this sort of replay attacks (i.e. replacing a page with an older
> copy of the page). That would probably require having some global
> checksum or something like that.

Uh, I think the only thing we could do is to add the page number into
the nonce so the page would have to be replaced in the same place in the
table, but it hardly seems worth it.

> > > Which seems pretty annoying, because then the checksums won't verify
> > > data as sent to the storage system, and verify checksums would require
> > > access to all keys (how do you do that in offline mode?).
> > 
> > Given the scheme above I don't see why that would be an issue. The keys
> > are all accessible via the MDEK, which is in turn available via the KEK.
> > 
> 
> I just don't know how the offline tools will access the KMS to get the
> keys. But maybe that's not an issue. But even then I think it's kinda
> against the idea of checksums that they would not checksum what was sent
> to the storage system.

Oh, I see your point now.  pgchecksum will look at the page and think it
is corrupt.  It would need access to the keys to verify it, and only for
whole-cluster encryption or if there is a per-page flag (it can't easily
do system table lookups).

The crazy seems more sane now --- "encrypt the page with CRC contents as
zero" (which we probably already do to compute the CRC), then compute
the CRC, and modify the page CRC.

I kind of feel we need to decide this now so our tooling can plan for it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 13, 2019 at 07:34:22AM -0400, Joe Conway wrote:
> I stand by my position. At a minimum, we need a choice of AES128 and AES256.

These are compelling arguments.  Agreed.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote:
>On Mon, Jul 15, 2019 at 03:47:59AM +0200, Tomas Vondra wrote:
>> On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote:
>> > We could check the CRC prior to encryption and throw an ERROR if it is
>> > not correct. After decryption we can check it again -- if it no longer
>> > matches we would know there way a corruption or change of the
>> > ciphertext, no?
>> >
>> > Hmm, I guess the entire page of ciphertext could be faked including CRC,
>> > so this would only really cover corruption, not an intentional change if
>> > it were done properly.
>> >
>>
>> I don't think any of the schemes discussed here provides protection
>> against this sort of replay attacks (i.e. replacing a page with an older
>> copy of the page). That would probably require having some global
>> checksum or something like that.
>
>Uh, I think the only thing we could do is to add the page number into
>the nonce so the page would have to be replaced in the same place in the
>table, but it hardly seems worth it.
>
>> > > Which seems pretty annoying, because then the checksums won't verify
>> > > data as sent to the storage system, and verify checksums would require
>> > > access to all keys (how do you do that in offline mode?).
>> >
>> > Given the scheme above I don't see why that would be an issue. The keys
>> > are all accessible via the MDEK, which is in turn available via the KEK.
>> >
>>
>> I just don't know how the offline tools will access the KMS to get the
>> keys. But maybe that's not an issue. But even then I think it's kinda
>> against the idea of checksums that they would not checksum what was sent
>> to the storage system.
>
>Oh, I see your point now.  pgchecksum will look at the page and think it
>is corrupt.  It would need access to the keys to verify it, and only for
>whole-cluster encryption or if there is a per-page flag (it can't easily
>do system table lookups).
>
>The crazy seems more sane now --- "encrypt the page with CRC contents as
>zero" (which we probably already do to compute the CRC), then compute
>the CRC, and modify the page CRC.
>

Huh? So you want to

1) set CRC to 0
2) encrypt the page
3) compute CRC
4) set CRC to value computed in (3)
5) encrypt the page again

That seems pretty awful from performance POV, and it does not really
solve much as we'd still need to decrypt the page while verifying the
checksums (because the CRC is in the page header, which is encrypted).

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
>On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
>> One extra thing we should consider is authenticated encryption. We can't
>> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> as that does not provide integrity protection (i.e. can't detect when
>> the ciphertext was corrupted due to disk failure or intentionally). And
>> we can't quite rely on checksums, because that checksums the plaintext
>> and is stored encrypted.
>
>Uh, if someone modifies a few bytes of the page, we will decrypt it, but
>the checksum (per-page or WAL) will not match our decrypted output.  How
>would they make it match the checksum without already knowing the key.
>I read [1] but could not see that explained.
>

Our checksum is only 16 bits, so perhaps one way would be to just
generate 64k of randomly modified pages and hope one of them happens to
hit the right checksum value. Not sure how practical such attack is, but
it does require just filesystem access.

FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
a cryptographic hash algorithm. Now, maybe we don't want authenticated
encryption (e.g. XTS is not authenticated, unlike GCM/CCM).

>This post discussed it:
>
>    https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac
>
>I realize in a new system we might prefer encrypt-then-mac, TLS and SSL
>do it differently, and I don't think the security problems of
>MAC-then-Encrypt apply to our use-case, e.g. API programming errors.
>
>If we want to go crazy, we could encrypt, assume zeros for the CRC,
>compute the MAC and put it in the place of the CRC is, but then tools
>that read CRC would see that as an error, so we don't want to go there.
>Yes, crazy.
>
>> Which seems pretty annoying, because then the checksums won't verify
>> data as sent to the storage system, and verify checksums would require
>> access to all keys (how do you do that in offline mode?).
>
>Uh, the keys are stored in a PGDATA file --- seems simple enough, but we
>would either have to do whole-cluster encryption or have some per-page
>encryption flag.
>

And how do you know which files are encrypted and which are not, and
which keys are used for which file? Presumably that's in some system
catalog, which is not available in offline mode. 

>> But the main issue with checksum-then-encrypt is it's essentially
>> "MAC-then-Encrypt" and that does not provide Authenticated Encryption
>> security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
>> in which case we'll need to store the MAC somewhere (probably in the
>> same place as the nonce/IV/key/... for each page).
>
>I don't think we are planning to store the nonce/IV on each page but
>rather use the LSN (already on the page), and perhaps in addition, the
>page number.

But the LSN is in the page header, and AFAICS the page header is
encrypted. So how do you decrypt the page without knowing the LSN (which
I think you need to know in otder to derive the IV)?

Also, we probably don't want to expose the checksum, because it may
reveal information about page contents (since it's not a HMAC).


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote:
> On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote:
> > The crazy seems more sane now --- "encrypt the page with CRC contents as
> > zero" (which we probably already do to compute the CRC), then compute
> > the CRC, and modify the page CRC.
> > 
> 
> Huh? So you want to
> 
> 1) set CRC to 0
> 2) encrypt the page
> 3) compute CRC
> 4) set CRC to value computed in (3)
> 5) encrypt the page again
> 
> That seems pretty awful from performance POV, and it does not really
> solve much as we'd still need to decrypt the page while verifying the
> checksums (because the CRC is in the page header, which is encrypted).

No, I was thinking we would overwrite whatever the encrypted output was
in the spot that has the CRC with the computed CRC.  Yeah, sounds even
crazier now that I said it --- never mind.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Hi,

Some more thoughts on CBC vs CTR modes. There are a number of
advantages to using CTR mode for page encryption.

CTR encryption modes can be fully parallelized, whereas CBC can only
parallelized for decryption. While both can use AES specific hardware
such as AES-NI, CTR modes can go a step further and use vectorized
instructions.

On an i7-8559U (with AES-NI) I get a 4x speed improvement for
CTR-based modes vs CBC when run on 8K of data:

# openssl speed -evp ${cipher}
type             16 bytes     64 bytes    256 bytes   1024 bytes
8192 bytes  16384 bytes
aes-128-cbc    1024361.51k  1521249.60k  1562033.41k  1571663.87k
1574537.90k  1575512.75k
aes-128-ctr     696866.85k  2214441.86k  4364903.85k  5896221.35k
6559735.81k  6619594.75k
aes-128-gcm     642758.92k  1638619.09k  3212068.27k  5085193.22k
6366035.97k  6474006.53k
aes-256-cbc     940906.25k  1114628.44k  1131255.13k  1138385.92k
1140258.13k  1143592.28k
aes-256-ctr     582161.82k  1896409.32k  3216926.12k  4249708.20k
4680299.86k  4706375.00k
aes-256-gcm     553513.89k  1532556.16k  2705510.57k  3931744.94k
4615812.44k  4673093.63k

For relation data where the encryption is going to be per page,
there's flexibility in how the CTR nonce (IV + counter) is generated.
With an 8K page, the counter need only go up to 512 for each page
(8192-bytes per page / 16-bytes per AES-block). That would require
9-bits for the counter. Rounding that up to 16-bits allows for wider
pages and it still uses only two bytes of the counter while ensuring
that it'd be unique per AES-block. The remaining 14-bytes would be
populated with some other data that is guaranteed unique per
page-write to allow encryption via the same per-relation-file derived
key. From what I gather, the LSN is a candidate though it'd have to be
stored in plaintext for decryption.

What's important is that writing the two pages (either different
locations or the same page back again) never reuses the same nonce
with the same key. Using the same nonce with a different key is fine.

With any of these schemes the same inputs will generate the same
outputs. With CTR mode for WAL this would be an issue if the same key
and deterministic nonce (ex: LSN + offset) is reused in multiple
places. That does not have to be the same cluster either. For example
if two replicas are promoted from the same backup with the same master
key, they would generate the same WAL CTR stream, reusing the
key/nonce pair. Ditto for starting off with a master key and deriving
per-relation keys in a cloned installation off some deterministic
attribute such as oid.

This can be avoided by deriving new keys per file (not just per
relation) from a random salt. It'd be stored out of band and combined
with the master key to derive the specific key used for that CTR
stream. If there's a desire for supporting multiple ciphers or key
sizes, that could be stored alongside the salt. Perhaps use the same
location or lack of it to indicate "not encrypted" as well.

Per-file salts and derived keys would facilitate re-keying a table
piecemeal, file by file, by generating a new salt/derived-key,
encrypting a copy of the decrypted file, and doing an atomic rename.
The files contents would change but its length and any references to
pages or byte offsets would stay valid. (I think this would work for
CBC modes too as there's nothing CTR specific about it.)

I'm not sure of is how to handle randomizing the relation file IV in a
cloned database. Until the key for a relation file or segment is
rotated it'd have the same deterministic IV generated as its source as
the LSN would continue from the same point. One idea is with 128-bits
for the IV, one could have 64-bits for LSN, 16-bits for AES-block
counter, and the remaining 48-bits be randomized; though you'd need to
store those 48-bits somewhere per-page (basically it's a salt per
page). That'd give some protection from the clone's new data be
encrypted with the same stream as the parent's. Another option would
be to track ranges of LSNs and have a centralized list of 48-bit
randomized salts. That would remove the need for additional salt per
page though you'd have to do a lookup on that shared list to figure
out which to use.

CTR mode is definitely more complicated than a pure random-IV + CBC
but with any deterministic generation of IVs for CBC mode you're going
to have some of these same problems as well.

Regarding CRCs, CTR mode has the advantage of not destroying the rest
of the stream to replace the CRC bytes. With CBC mode any change would
cascade and corrupt the rest of data the down stream from that block.
With CTR mode you can overwrite the CRC's location with the CRC or a
truncated MAC of the encrypted data as each byte is encrypted
separately. At decryption time you simply ignore the decrypted output
of those bytes and zero them out again. A CRC of encrypted data (but
not a partial MAC) could be checked offline without access to the key.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote:
> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
> > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
> > > One extra thing we should consider is authenticated encryption. We can't
> > > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> > > as that does not provide integrity protection (i.e. can't detect when
> > > the ciphertext was corrupted due to disk failure or intentionally). And
> > > we can't quite rely on checksums, because that checksums the plaintext
> > > and is stored encrypted.
> > 
> > Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> > the checksum (per-page or WAL) will not match our decrypted output.  How
> > would they make it match the checksum without already knowing the key.
> > I read [1] but could not see that explained.
> > 
> 
> Our checksum is only 16 bits, so perhaps one way would be to just
> generate 64k of randomly modified pages and hope one of them happens to
> hit the right checksum value. Not sure how practical such attack is, but
> it does require just filesystem access.

Yes, that would work, and opens the question of whether our checksum is
big enough for this, and if it is not, we need to find space for it,
probably with a custom encrypted page format.  :-(   And that makes
adding encryption offline almost impossible because you potentially have
to move tuples around.  Yuck!

> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
> a cryptographic hash algorithm. Now, maybe we don't want authenticated
> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).

I thought just encrypting the CRC value would be enough to detect
changes, but you are right that some you could just do 64k pages until
one hit.

> > This post discussed it:
> > 
> >     https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac
> > 
> > I realize in a new system we might prefer encrypt-then-mac, TLS and SSL
> > do it differently, and I don't think the security problems of
> > MAC-then-Encrypt apply to our use-case, e.g. API programming errors.
> > 
> > If we want to go crazy, we could encrypt, assume zeros for the CRC,
> > compute the MAC and put it in the place of the CRC is, but then tools
> > that read CRC would see that as an error, so we don't want to go there.
> > Yes, crazy.
> > 
> > > Which seems pretty annoying, because then the checksums won't verify
> > > data as sent to the storage system, and verify checksums would require
> > > access to all keys (how do you do that in offline mode?).
> > 
> > Uh, the keys are stored in a PGDATA file --- seems simple enough, but we
> > would either have to do whole-cluster encryption or have some per-page
> > encryption flag.
> > 
> 
> And how do you know which files are encrypted and which are not, and
> which keys are used for which file? Presumably that's in some system
> catalog, which is not available in offline mode.

You would need either all-cluster encryption (no need to check) or a
per-page bit that says the page is encrypted, and the bit has to be in
the part of the page that is not encryped, e.g., near LSN.

> > > But the main issue with checksum-then-encrypt is it's essentially
> > > "MAC-then-Encrypt" and that does not provide Authenticated Encryption
> > > security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
> > > in which case we'll need to store the MAC somewhere (probably in the
> > > same place as the nonce/IV/key/... for each page).
> > 
> > I don't think we are planning to store the nonce/IV on each page but
> > rather use the LSN (already on the page), and perhaps in addition, the
> > page number.
> 
> But the LSN is in the page header, and AFAICS the page header is
> encrypted. So how do you decrypt the page without knowing the LSN (which
> I think you need to know in otder to derive the IV)?

My poposal was that the first 16 bytes of the page are not encrypted.

> Also, we probably don't want to expose the checksum, because it may
> reveal information about page contents (since it's not a HMAC).

Uh, I have not heard of that as an issue.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Jul-15, Bruce Momjian wrote:

> My point is that doing encryption of only some data might actually make
> the system slower due to the lookups, so I think we need to implement
> all-cluster encryption and then see what the overhead is, and if there
> are use-cases for not encrypting only some data.

We can keep the keys in the relcache.  It doesn't have to be slow.  It
is certainly slower to have to encrypt *all* data, which can be
massively larger than the sensitive portion of the database.

If we need the keys for offline operation (where relcache is not
reachable), we can keep pointers to the key files in the filesystem --
for example for an encrypted table we would keep a new file, say
<relfilenode>.key, which could be a symlink to the encrypted key file.
The tool already has access to the key data, but the symlink lets it
know *which* key to use; random onlookers cannot get the key data
because the file is encrypted with the master key.

Any table without the key file is assumed to be unencrypted.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote:
>On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote:
>> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
>> > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
>> > > One extra thing we should consider is authenticated encryption. We can't
>> > > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> > > as that does not provide integrity protection (i.e. can't detect when
>> > > the ciphertext was corrupted due to disk failure or intentionally). And
>> > > we can't quite rely on checksums, because that checksums the plaintext
>> > > and is stored encrypted.
>> >
>> > Uh, if someone modifies a few bytes of the page, we will decrypt it, but
>> > the checksum (per-page or WAL) will not match our decrypted output.  How
>> > would they make it match the checksum without already knowing the key.
>> > I read [1] but could not see that explained.
>> >
>>
>> Our checksum is only 16 bits, so perhaps one way would be to just
>> generate 64k of randomly modified pages and hope one of them happens to
>> hit the right checksum value. Not sure how practical such attack is, but
>> it does require just filesystem access.
>
>Yes, that would work, and opens the question of whether our checksum is
>big enough for this, and if it is not, we need to find space for it,
>probably with a custom encrypted page format.  :-(   And that makes
>adding encryption offline almost impossible because you potentially have
>to move tuples around.  Yuck!
>

Right. We've been working on allowing to disable checksum online, and it
would be useful to allow something like that for encryption too I guess.
And without some sort of page-level flag that won't be possible, which
would be rather annoying.

Not sure it needs to be in the page itself, though - that's pretty much
why I proposed to store metadata (IV, key ID, ...) for encryption in a
new fork. That would be a bit more flexible than storing it in the page
itself (e.g. different encryption schemes might easily store different
amounts of metadata).

Maybe a new fork is way too complex solution, not sure.

>> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
>> a cryptographic hash algorithm. Now, maybe we don't want authenticated
>> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).
>
>I thought just encrypting the CRC value would be enough to detect
>changes, but you are right that some you could just do 64k pages until
>one hit.
>

Right. Not sure that's really a practical attack we need to worry about,
considering all of this is vulnerable to replay attacks.

>> > This post discussed it:
>> >
>> >     https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac
>> >
>> > I realize in a new system we might prefer encrypt-then-mac, TLS and SSL
>> > do it differently, and I don't think the security problems of
>> > MAC-then-Encrypt apply to our use-case, e.g. API programming errors.
>> >
>> > If we want to go crazy, we could encrypt, assume zeros for the CRC,
>> > compute the MAC and put it in the place of the CRC is, but then tools
>> > that read CRC would see that as an error, so we don't want to go there.
>> > Yes, crazy.
>> >
>> > > Which seems pretty annoying, because then the checksums won't verify
>> > > data as sent to the storage system, and verify checksums would require
>> > > access to all keys (how do you do that in offline mode?).
>> >
>> > Uh, the keys are stored in a PGDATA file --- seems simple enough, but we
>> > would either have to do whole-cluster encryption or have some per-page
>> > encryption flag.
>> >
>>
>> And how do you know which files are encrypted and which are not, and
>> which keys are used for which file? Presumably that's in some system
>> catalog, which is not available in offline mode.
>
>You would need either all-cluster encryption (no need to check) or a
>per-page bit that says the page is encrypted, and the bit has to be in
>the part of the page that is not encryped, e.g., near LSN.
>
>> > > But the main issue with checksum-then-encrypt is it's essentially
>> > > "MAC-then-Encrypt" and that does not provide Authenticated Encryption
>> > > security - see [1]. We should be looking at "Encrypt-then-MAC" instead,
>> > > in which case we'll need to store the MAC somewhere (probably in the
>> > > same place as the nonce/IV/key/... for each page).
>> >
>> > I don't think we are planning to store the nonce/IV on each page but
>> > rather use the LSN (already on the page), and perhaps in addition, the
>> > page number.
>>
>> But the LSN is in the page header, and AFAICS the page header is
>> encrypted. So how do you decrypt the page without knowing the LSN (which
>> I think you need to know in otder to derive the IV)?
>
>My poposal was that the first 16 bytes of the page are not encrypted.
>

Ah, I see.

>> Also, we probably don't want to expose the checksum, because it may
>> reveal information about page contents (since it's not a HMAC).
>
>Uh, I have not heard of that as an issue.
>

To clarify, I think it's more a general issue - the checksum does leak a
bit of information about the plaintext, I think that's fairly obvious. I
don't know if 16 bits is enough for practical attacks, though.

But it clearly is not the same thing as HMAC, so we should not treat it
as such.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote:
>On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote:
>> On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote:
>> > The crazy seems more sane now --- "encrypt the page with CRC contents as
>> > zero" (which we probably already do to compute the CRC), then compute
>> > the CRC, and modify the page CRC.
>> >
>>
>> Huh? So you want to
>>
>> 1) set CRC to 0
>> 2) encrypt the page
>> 3) compute CRC
>> 4) set CRC to value computed in (3)
>> 5) encrypt the page again
>>
>> That seems pretty awful from performance POV, and it does not really
>> solve much as we'd still need to decrypt the page while verifying the
>> checksums (because the CRC is in the page header, which is encrypted).
>
>No, I was thinking we would overwrite whatever the encrypted output was
>in the spot that has the CRC with the computed CRC.  Yeah, sounds even
>crazier now that I said it --- never mind.
>

Uh, how could that possibly work? Symmetric ciphers are "diffusing" the
bits within the block, i.e. replacing 16 bits in a 128-bit ciphertext
block will affect the whole plaintext block, not just the matching 16
bits of plaintext.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 

On Tue, Jul 16, 2019 at 02:04:58AM +0200, Tomas Vondra wrote:
> On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote:
> > On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote:
> > > On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote:
> > > > The crazy seems more sane now --- "encrypt the page with CRC contents as
> > > > zero" (which we probably already do to compute the CRC), then compute
> > > > the CRC, and modify the page CRC.
> > > >
> > > 
> > > Huh? So you want to
> > > 
> > > 1) set CRC to 0
> > > 2) encrypt the page
> > > 3) compute CRC
> > > 4) set CRC to value computed in (3)
> > > 5) encrypt the page again
> > > 
> > > That seems pretty awful from performance POV, and it does not really
> > > solve much as we'd still need to decrypt the page while verifying the
> > > checksums (because the CRC is in the page header, which is encrypted).
> > 
> > No, I was thinking we would overwrite whatever the encrypted output was
> > in the spot that has the CRC with the computed CRC.  Yeah, sounds even
> > crazier now that I said it --- never mind.
> > 
> 
> Uh, how could that possibly work? Symmetric ciphers are "diffusing" the
> bits within the block, i.e. replacing 16 bits in a 128-bit ciphertext
> block will affect the whole plaintext block, not just the matching 16
> bits of plaintext.

Yes, it would only work if the checksum was the last part of the page,
or if we used CTR mode, where changing the source bits doens't affect
the later bits.  I am thinking crazy here, I know, but it seemed worth
mentioning in case someone liked it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote:
> Hi,
> 
> Some more thoughts on CBC vs CTR modes. There are a number of
> advantages to using CTR mode for page encryption.
> 
> CTR encryption modes can be fully parallelized, whereas CBC can only
> parallelized for decryption. While both can use AES specific hardware
> such as AES-NI, CTR modes can go a step further and use vectorized
> instructions.
> 
> On an i7-8559U (with AES-NI) I get a 4x speed improvement for
> CTR-based modes vs CBC when run on 8K of data:
> 
> # openssl speed -evp ${cipher}
> type             16 bytes     64 bytes    256 bytes   1024 bytes
> 8192 bytes  16384 bytes
> aes-128-cbc    1024361.51k  1521249.60k  1562033.41k  1571663.87k
> 1574537.90k  1575512.75k
> aes-128-ctr     696866.85k  2214441.86k  4364903.85k  5896221.35k
> 6559735.81k  6619594.75k
> aes-128-gcm     642758.92k  1638619.09k  3212068.27k  5085193.22k
> 6366035.97k  6474006.53k
> aes-256-cbc     940906.25k  1114628.44k  1131255.13k  1138385.92k
> 1140258.13k  1143592.28k
> aes-256-ctr     582161.82k  1896409.32k  3216926.12k  4249708.20k
> 4680299.86k  4706375.00k
> aes-256-gcm     553513.89k  1532556.16k  2705510.57k  3931744.94k
> 4615812.44k  4673093.63k

Wow, I am seeing CTR as 2x faster here too.

> For relation data where the encryption is going to be per page,
> there's flexibility in how the CTR nonce (IV + counter) is generated.
> With an 8K page, the counter need only go up to 512 for each page
> (8192-bytes per page / 16-bytes per AES-block). That would require
> 9-bits for the counter. Rounding that up to 16-bits allows for wider
> pages and it still uses only two bytes of the counter while ensuring
> that it'd be unique per AES-block. The remaining 14-bytes would be
> populated with some other data that is guaranteed unique per
> page-write to allow encryption via the same per-relation-file derived
> key. From what I gather, the LSN is a candidate though it'd have to be
> stored in plaintext for decryption.

Oh, for CTR, we need to increment the counter for each 16-byte block ---
got it.

> What's important is that writing the two pages (either different
> locations or the same page back again) never reuses the same nonce
> with the same key. Using the same nonce with a different key is fine.

Uh, I think we can use LSN and page-number to be unique.

> With any of these schemes the same inputs will generate the same
> outputs. With CTR mode for WAL this would be an issue if the same key
> and deterministic nonce (ex: LSN + offset) is reused in multiple
> places. That does not have to be the same cluster either. For example

Very good point, since CTR does not use the user data as part of the
later encryption.

> if two replicas are promoted from the same backup with the same master
> key, they would generate the same WAL CTR stream, reusing the
> key/nonce pair. Ditto for starting off with a master key and deriving
> per-relation keys in a cloned installation off some deterministic
> attribute such as oid.

Uh, when we promote a standby, don't we increment the timeline?  Does
that help?  I don't know what we could use to distingish two standbys
that are both promoted and using the same key --- there is nothing
unique about them.

> This can be avoided by deriving new keys per file (not just per
> relation) from a random salt. It'd be stored out of band and combined
> with the master key to derive the specific key used for that CTR
> stream. If there's a desire for supporting multiple ciphers or key
> sizes, that could be stored alongside the salt. Perhaps use the same
> location or lack of it to indicate "not encrypted" as well.

You mean the cluster would have its own random key?  Unfortunately all
clusters in a replica set have the same Database system identifier as
the primary.

> Per-file salts and derived keys would facilitate re-keying a table
> piecemeal, file by file, by generating a new salt/derived-key,
> encrypting a copy of the decrypted file, and doing an atomic rename.
> The files contents would change but its length and any references to
> pages or byte offsets would stay valid. (I think this would work for
> CBC modes too as there's nothing CTR specific about it.)

Storing that might be a problem, particularly to access during crash
recovery.

> I'm not sure of is how to handle randomizing the relation file IV in a
> cloned database. Until the key for a relation file or segment is
> rotated it'd have the same deterministic IV generated as its source as
> the LSN would continue from the same point. One idea is with 128-bits
> for the IV, one could have 64-bits for LSN, 16-bits for AES-block
> counter, and the remaining 48-bits be randomized; though you'd need to
> store those 48-bits somewhere per-page (basically it's a salt per
> page). That'd give some protection from the clone's new data be
> encrypted with the same stream as the parent's. Another option would
> be to track ranges of LSNs and have a centralized list of 48-bit
> randomized salts. That would remove the need for additional salt per
> page though you'd have to do a lookup on that shared list to figure
> out which to use.
> 
> CTR mode is definitely more complicated than a pure random-IV + CBC
> but with any deterministic generation of IVs for CBC mode you're going
> to have some of these same problems as well.

This is starting to sound unworkable for our usecase.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Fri, Jul 12, 2019 at 02:15:02PM +0900, Masahiko Sawada wrote:
> > > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> > > 8k pages will use the LSN as a nonce, which will be encrypted to
> > > generate the initialization vector (IV).  We will not encrypt the first
> > > 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> > > use the WAL file segment number as the nonce and the IV will be created
> > > in the same way.
> > >
> > > wal_log_hints will be enabled automatically in encryption mode, like we
> > > do for checksum mode, so we never encrypt different 8k pages with the
> > > same IV.
> >
> > I guess that different two pages can have the same LSN when a heap
> > update modifies both a page for old tuple and another page for new
> > tuple.
> >
> > heapam.c:3707
> >         recptr = log_heap_update(relation, buffer,
> >                                  newbuf, &oldtup, heaptup,
> >                                  old_key_tuple,
> >                                  all_visible_cleared,
> >                                  all_visible_cleared_new);
> >         if (newbuf != buffer)
> >         {
> >             PageSetLSN(BufferGetPage(newbuf), recptr);
> >         }
> >         PageSetLSN(BufferGetPage(buffer), recptr);
> >
> > Wouldn't it a problem?
>
> I had the same question.  If someone does:
>
>         UPDATE tab SET col = col + 1
>
> then each row change gets its own LSN.  You are asking if an update that
> just expires one row and adds it to a new page gets the same LSN.  I
> don't know.

The following scripts can reproduce that different two pages have the same LSN.

=# create table test (a int);
CREATE TABLE
=# insert into test select generate_series(1, 226);
INSERT 0 226
=# update test set a = a where a = 1;
UPDATE 1
=# select lsn from page_header(get_raw_page('test', 0));
    lsn
-----------
 0/1690488
(1 row)

=# select lsn from page_header(get_raw_page('test', 1));
    lsn
-----------
 0/1690488
(1 row)

So I think it's better to use LSN and page number to create IV. If we
modify different tables by single WAL we also would need OID or
relfilenode but I don't think currently we have such operations.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Jul 15, 2019 at 9:38 PM Antonin Houska <ah@cybertec.at> wrote:
>
> Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > On Mon, Jun 17, 2019 at 11:02 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > > On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote:
> > > >On 6/17/19 8:29 AM, Masahiko Sawada wrote:
> > > >> From perspective of  cryptographic, I think the fine grained TDE would
> > > >> be better solution. Therefore if we eventually want the fine grained
> > > >> TDE I wonder if it might be better to develop the table/tablespace TDE
> > > >> first while keeping it simple as much as possible in v1, and then we
> > > >> can provide the functionality to encrypt other data in database
> > > >> cluster to satisfy the encrypting-everything requirement. I guess that
> > > >> it's easier to incrementally add encryption target objects rather than
> > > >> making it fine grained while not changing encryption target objects.
> > > >>
> > > >> FWIW I'm writing a draft patch of per tablespace TDE and will submit
> > > >> it in this month. We can more discuss the complexity of the proposed
> > > >> TDE using it.
> > > >
> > > >+1
> > > >
> > > >Looking forward to it.
> > > >
> > >
> > > Yep. In particular, I'm interested in those aspects:
> > >
> >
> > Attached the draft version patch sets of per tablespace transparent
> > data at rest encryption.
>

Thank you for your email

> I was worried that there's competition between us but now that I've checked
> your patch set I see that you already use some parts of
>
> https://commitfest.postgresql.org/23/2104/
>
> although not the latest version. I'm supposed to work on the encryption now,
> so thinking what to do next. I think we should coordinate the effort, possibly
> off-list. The earlier we have a single patch set the more efficient the work
> should be.

Agreed. Let's discuss how we can coordinate the effort. I also think
it could be off-list as that's mostly about non-technical topic.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Jul 12, 2019 at 7:37 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote:
> > On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote:
> > > Please see my other reply (and
> > > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf
> > > appendix C as pointed out by Ryan downthread).
> > >
> > > At least in my mind, I trust a published specification from the
> > > nation-state level over random blogs or wikipedia. If we can find some
> > > equivalent published standards that contradict NIST we should discuss
> > > it, but for my money I would prefer to stick with the NIST recommended
> > > method to produce the IVs.
> >
> > So, we have had a flurry of activity on this thread in the past day, so
> > let me summarize:
>
> Seems we have an updated approach:
>
> First, we need to store the symmetric encryption key in the data
> directory, like we do for SSL certificates and private keys.  (Crash
> recovery needs access to this key, so we can't easily store it in a
> database table.)  We will pattern it after the GUC
> ssl_passphrase_command.   We will need to decide on a format for the
> symmetric encryption key in the file so we can check that the supplied
> passphrase properly unlocks the key.
>
> Our first implementation will encrypt the entire cluster.  We can later
> consider encryption per table or tablespace.  It is unclear if
> encrypting different parts of the system with different keys is useful
> or feasible.  (This is separate from key rotation.)
>
> We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL.
> 8k pages will use the LSN as a nonce, which will be encrypted to
> generate the initialization vector (IV).  We will not encrypt the first
> 16 bytes of each pages so the LSN can be used in this way.  The WAL will
> use the WAL file segment number as the nonce and the IV will be created
> in the same way.
>
> wal_log_hints will be enabled automatically in encryption mode, like we
> do for checksum mode, so we never encrypt different 8k pages with the
> same IV.
>
> There will need to be a pg_control field to indicate that encryption is
> in use.
>
> Right now we don't support the online changing of a cluster's checksum
> mode, so I suggest we create a utility like pg_checksums --enable to
> allow offline key rotation.  Once we get online checksum mode changing
> ability, we can look into use that for encryption key rotation.
>

I've re-considered the design of TDE feature based on the discussion
so far. The one of the main open question is the granular of
encryption objects: cluster encryption or more-granular-than-cluster
encryption. The followings describe about the new TDE design when we
choose table-level encryption or something-new-group-level encryption.

General
========
We will use AES and support both AES-128 and AES-256. User can specify
the new initdb option something like --aes-128 or --aes-256 to enable
encryption and must specify --encryption-key-passphrase-command along
with. (I guess we also require openssl library.) If these options are
specified, we write the key length to the control file and derive the
KEK and generate MDEK during initdb. wal_log_hints will be enabled
automatically in encryption mode, like we do for checksum mode,

Key Management
==============
We will use 3-tier key architecture as Joe proposed.

  1. A master key encryption key (KEK): this is the ley supplied by the
     database admin using something akin to ssl_passphrase_command

  2. A master data encryption key (MDEK): this is a generated key using a
     cryptographically secure pseudo-random number generator. It is
     encrypted using the KEK, probably with Key Wrap (KW):
     or maybe better Key Wrap with Padding (KWP):

  3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
      table specific keys.

  3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
      generate new keys when needed for WAL.

We store MDEK to the plain file (say global/pgkey) after encrypted
with the KEK. I might want to store the hash of passphrase of the KEK
in order to verify the correctness of the given passphrase. However we
don't need to store TDEK and WDEK as we can derive them as needed. The
key file can be read by both backend processes and front-end tools.

When postmaster startup, it reads the key file and decrypts MDEK and
derive WDEK using key id for WDEK. WDEK is loaded to the key hash map
(keyid -> key) on the shared memory. Also we derive TDEK as needed
when reading tables or indexes and add it to the key hash map as well
if not exists.

Buffer Encryption
==============
We will use AES-CBC for buffer encryption. We will add key id (4byte)
to after the pd_lsn(8byte) in PageHeaderData and we will not encrypt
first 16 byte of each pages so the LSN and key id can be used. We can
store an invalid key id to tell us that the table is not encrypted.
There two benefits of storing key id to the page header: offline tools
can get key id (and know the table is encrypted or not) and it's
helpful for online rekey in the future.

I've considered to store IV and key id to a new fork but I felt that
it is complex because we will always need to have the fork on the
shared buffer when any pages of its main fork is written to the disk.
If almost buffers of the shared buffers are dirtied and theirs new
forks are not  loaded to the shared buffer, we might need to load the
new fork and write the page to the disk and then evict some pages,
over and over.

We will use (page lsn, page number) to create a nonce. IVs are created
by encrypting the nonce with its TDEK.

WAL Encryption
=============
We will use AES-CTR for WAL encryption and encrypt each WAL pages with WDEK.

We will use WAL segment number to create a nonce. Similar to buffer
encryption, IVs are created using by the nonce and WDEK.

If we want to support enabling or disabling encryption after initdb we
might want to have key id in the WAL page header.

Front-end Tool Support
==================
We will add --encryption-key-passphrase-command option to the
front-end tools that read database files or WAL segment files directly.
They can get KEK via --encryption-key-passphrase-command and get MDEK
by reading the key file. Also they can know the key length by checking
the control file. Since they can derive TDEK using by key id stored in
the page header they can decrypt database files. Similarly, they also
can decrypt WAL as they can know the key id of WDEK.

Master Key Rotation
================
We will support new command-line tool that rotates the master key
offline. It accepts --old-encryption-key-passphrase-command option and
--new-encryption-key-passphrase-command to get old KEK and new KEK
respectively. It decrypt MDEK with the old key and encrypt it with
the new key.

There is concern about the performance overhead by both looking up
keys and checking if the object is encrypted or not but with this
design we can know them by reading page and by checking hash map on
the shared memory. It works fine unless we have to have a huge number
of keys in the hash map. So I guess the overhead doesn't become
obvious. In addition, this key management design is similar to the PoC
patch I created before and evaluated its the performance overhead a few
month ago. In the evaluation, I didn't see such overhead. See [1].

[1]
https://www.slideshare.net/masahikosawada98/transparent-data-encryptoin-in-postgresql-and-integratino-with-key-management-service/31




Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
> >> One extra thing we should consider is authenticated encryption. We can't
> >> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> >> as that does not provide integrity protection (i.e. can't detect when
> >> the ciphertext was corrupted due to disk failure or intentionally). And
> >> we can't quite rely on checksums, because that checksums the plaintext
> >> and is stored encrypted.
> >
> >Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> >the checksum (per-page or WAL) will not match our decrypted output.  How
> >would they make it match the checksum without already knowing the key.
> >I read [1] but could not see that explained.
> >
> 
> Our checksum is only 16 bits, so perhaps one way would be to just
> generate 64k of randomly modified pages and hope one of them happens to
> hit the right checksum value. Not sure how practical such attack is, but
> it does require just filesystem access.

I don't think you can easily generate 64k of different checksums this way. If
the data is random, I suppose that each set of 2^(128 - 16) blocks will
contain the the same checksum after decryption. Thus even you generate 64k of
different ciphertext blocks that contain the checksum, some (many?)  checksums
will be duplicate. Unfortunately the math to describe this problem does not
seem to be trivial.

Also note that if you try to generate ciphertext, decryption of which will
result in particular value of checksum, you can hardly control the other 14
bytes of the block, which in turn are used to verify the checksum.

> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
> a cryptographic hash algorithm. Now, maybe we don't want authenticated
> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).

I'm also not sure if we should try to guarantee data authenticity /
integrity. As someone already mentioned elsewhere, page MAC does not help if
the whole page is replaced. (An extreme case is that old filesystem snapshot
containing the whole data directory is restored, although that will probably
make the database crash soon.)

We can guarantee integrity and authenticity of backup, but that's a separate
feature: someone may need this although it's o.k. for him to run the cluster
unencrypted.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

> On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote:
> >On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote:
> >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
> >> > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
> >> > > One extra thing we should consider is authenticated encryption. We can't
> >> > > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> >> > > as that does not provide integrity protection (i.e. can't detect when
> >> > > the ciphertext was corrupted due to disk failure or intentionally). And
> >> > > we can't quite rely on checksums, because that checksums the plaintext
> >> > > and is stored encrypted.
> >> >
> >> > Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> >> > the checksum (per-page or WAL) will not match our decrypted output.  How
> >> > would they make it match the checksum without already knowing the key.
> >> > I read [1] but could not see that explained.
> >> >
> >>
> >> Our checksum is only 16 bits, so perhaps one way would be to just
> >> generate 64k of randomly modified pages and hope one of them happens to
> >> hit the right checksum value. Not sure how practical such attack is, but
> >> it does require just filesystem access.
> >
> >Yes, that would work, and opens the question of whether our checksum is
> >big enough for this, and if it is not, we need to find space for it,
> >probably with a custom encrypted page format.  :-(   And that makes
> >adding encryption offline almost impossible because you potentially have
> >to move tuples around.  Yuck!
> >
>
> Right. We've been working on allowing to disable checksum online, and it
> would be useful to allow something like that for encryption too I guess.
> And without some sort of page-level flag that won't be possible, which
> would be rather annoying.
>
> Not sure it needs to be in the page itself, though - that's pretty much
> why I proposed to store metadata (IV, key ID, ...) for encryption in a
> new fork. That would be a bit more flexible than storing it in the page
> itself (e.g. different encryption schemes might easily store different
> amounts of metadata).
>
> Maybe a new fork is way too complex solution, not sure.

One problem of this new fork would be that contents of its buffer (the MAC
values) is not determined until the corresponding buffers of the MAIN fork get
encrypted. However encryption is performed by the storage layer (md.c), which
is not expected to lock other buffers (such as those of the "MAC fork"), read
their pages from disk or insert their WAL records.

This is different from the FSM or VM forks whose buffers are only updated
above the storage layer.

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote:
>Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>
>> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
>> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
>> >> One extra thing we should consider is authenticated encryption. We can't
>> >> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> >> as that does not provide integrity protection (i.e. can't detect when
>> >> the ciphertext was corrupted due to disk failure or intentionally). And
>> >> we can't quite rely on checksums, because that checksums the plaintext
>> >> and is stored encrypted.
>> >
>> >Uh, if someone modifies a few bytes of the page, we will decrypt it, but
>> >the checksum (per-page or WAL) will not match our decrypted output.  How
>> >would they make it match the checksum without already knowing the key.
>> >I read [1] but could not see that explained.
>> >
>>
>> Our checksum is only 16 bits, so perhaps one way would be to just
>> generate 64k of randomly modified pages and hope one of them happens to
>> hit the right checksum value. Not sure how practical such attack is, but
>> it does require just filesystem access.
>
>I don't think you can easily generate 64k of different checksums this way. If
>the data is random, I suppose that each set of 2^(128 - 16) blocks will
>contain the the same checksum after decryption. Thus even you generate 64k of
>different ciphertext blocks that contain the checksum, some (many?)  checksums
>will be duplicate. Unfortunately the math to describe this problem does not
>seem to be trivial.
>

I'm not sure what's your point, or why you care about the 128 bits, but I
don't think the math is very complicated (and it's exactly the same with
or without encryption). The probability of checksum collision for randomly
modified page is 1/64k, so p=~0.00153%. So probability of *not* getting a
collision is (1-p)=99.9985%. So with N pages, the probability of no
collisions is pow((1-p),N) which behaves like this:

      N     pow((1-p),N)
    --------------------
    10000           85%
    20000           73%
    30000           63%
    46000           49%
    200000           4%

So with 1.6GB relation you have about 96% chance of a checksum collision.

>Also note that if you try to generate ciphertext, decryption of which will
>result in particular value of checksum, you can hardly control the other 14
>bytes of the block, which in turn are used to verify the checksum.
>

But we don't care about the 14 bytes. In fact, we want the page header
(which includes both the checksums and the other 14B in the block) to
remain unchanged - the attack only needs to modify the remaining parts of
the 8kB page in a way to generate the same checksum on the plaintext.

And that's not that hard to do, IMHO, because the header is stored at the
beginning of the page. So we can just randomly modify the last AES block
(last 16B on the page) to minimize the corruption to the last block.

Now, I'm not saying this attack is particularly practical - it would
generate a fair number of checkpoint failures before getting the first
collision. So it'd trigger quite a few alerts, I guess.

>> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
>> a cryptographic hash algorithm. Now, maybe we don't want authenticated
>> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).
>
>I'm also not sure if we should try to guarantee data authenticity /
>integrity. As someone already mentioned elsewhere, page MAC does not help if
>the whole page is replaced. (An extreme case is that old filesystem snapshot
>containing the whole data directory is restored, although that will probably
>make the database crash soon.)
>
>We can guarantee integrity and authenticity of backup, but that's a separate
>feature: someone may need this although it's o.k. for him to run the cluster
>unencrypted.
>

Yes, I do agree with that. I think attempts to guarantee data authenticity
and/or integrity at the page level is mostly futile (replay attacks are an
example of why). IMHO we should consider that to be outside the threat
model TDE is expected to address.

IMO a better way to handle authenticity/integrity would be based on WAL,
which is essentially an authoritative log of operations. We should be able
to parse WAL, deduce expected state (min LSN, checksums) for each page,
and validate the cluster state based on that.

I still think having to decrypt the page in order to verify a checksum
(because the header is part of the encrypted page, and is computed from
the plaintext version) is not great.

regards

>-- 
>Antonin Houska
>Web: https://www.cybertec-postgresql.com

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Jul 19, 2019 at 01:32:01PM +0200, Antonin Houska wrote:
>Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>
>> On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote:
>> >On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote:
>> >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
>> >> > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
>> >> > > One extra thing we should consider is authenticated encryption. We can't
>> >> > > just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> >> > > as that does not provide integrity protection (i.e. can't detect when
>> >> > > the ciphertext was corrupted due to disk failure or intentionally). And
>> >> > > we can't quite rely on checksums, because that checksums the plaintext
>> >> > > and is stored encrypted.
>> >> >
>> >> > Uh, if someone modifies a few bytes of the page, we will decrypt it, but
>> >> > the checksum (per-page or WAL) will not match our decrypted output.  How
>> >> > would they make it match the checksum without already knowing the key.
>> >> > I read [1] but could not see that explained.
>> >> >
>> >>
>> >> Our checksum is only 16 bits, so perhaps one way would be to just
>> >> generate 64k of randomly modified pages and hope one of them happens to
>> >> hit the right checksum value. Not sure how practical such attack is, but
>> >> it does require just filesystem access.
>> >
>> >Yes, that would work, and opens the question of whether our checksum is
>> >big enough for this, and if it is not, we need to find space for it,
>> >probably with a custom encrypted page format.  :-(   And that makes
>> >adding encryption offline almost impossible because you potentially have
>> >to move tuples around.  Yuck!
>> >
>>
>> Right. We've been working on allowing to disable checksum online, and it
>> would be useful to allow something like that for encryption too I guess.
>> And without some sort of page-level flag that won't be possible, which
>> would be rather annoying.
>>
>> Not sure it needs to be in the page itself, though - that's pretty much
>> why I proposed to store metadata (IV, key ID, ...) for encryption in a
>> new fork. That would be a bit more flexible than storing it in the page
>> itself (e.g. different encryption schemes might easily store different
>> amounts of metadata).
>>
>> Maybe a new fork is way too complex solution, not sure.
>
>One problem of this new fork would be that contents of its buffer (the MAC
>values) is not determined until the corresponding buffers of the MAIN fork get
>encrypted. However encryption is performed by the storage layer (md.c), which
>is not expected to lock other buffers (such as those of the "MAC fork"), read
>their pages from disk or insert their WAL records.
>
>This is different from the FSM or VM forks whose buffers are only updated
>above the storage layer.
>

Yes, that seems like a valid issue :-(


-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:

> On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote:
> >Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
> >
> >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
> >> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
> >> >> One extra thing we should consider is authenticated encryption. We can't
> >> >> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
> >> >> as that does not provide integrity protection (i.e. can't detect when
> >> >> the ciphertext was corrupted due to disk failure or intentionally). And
> >> >> we can't quite rely on checksums, because that checksums the plaintext
> >> >> and is stored encrypted.
> >> >
> >> >Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> >> >the checksum (per-page or WAL) will not match our decrypted output.  How
> >> >would they make it match the checksum without already knowing the key.
> >> >I read [1] but could not see that explained.
> >> >
> >>
> >> Our checksum is only 16 bits, so perhaps one way would be to just
> >> generate 64k of randomly modified pages and hope one of them happens to
> >> hit the right checksum value. Not sure how practical such attack is, but
> >> it does require just filesystem access.
> >
> >I don't think you can easily generate 64k of different checksums this way. If
> >the data is random, I suppose that each set of 2^(128 - 16) blocks will
> >contain the the same checksum after decryption. Thus even you generate 64k of
> >different ciphertext blocks that contain the checksum, some (many?)  checksums
> >will be duplicate. Unfortunately the math to describe this problem does not
> >seem to be trivial.
> >
>
> I'm not sure what's your point, or why you care about the 128 bits, but I
> don't think the math is very complicated (and it's exactly the same with
> or without encryption). The probability of checksum collision for randomly
> modified page is 1/64k, so p=~0.00153%. So probability of *not* getting a
> collision is (1-p)=99.9985%. So with N pages, the probability of no
> collisions is pow((1-p),N) which behaves like this:
>
>      N     pow((1-p),N)
>    --------------------
>    10000           85%
>    20000           73%
>    30000           63%
>    46000           49%
>    200000           4%
>
> So with 1.6GB relation you have about 96% chance of a checksum collision.

I thought your attack proposal was to find valid (encrypted) checksum for a
given encrypted page. Instead it seems that you were only trying to say that
it's not too hard to generate page with a valid checksum in general. Thus the
attacker can try to modify the ciphertext again and again in a way that is not
quite random, but the chance to pass the checksum verification may still be
relatively high.

> >Also note that if you try to generate ciphertext, decryption of which will
> >result in particular value of checksum, you can hardly control the other 14
> >bytes of the block, which in turn are used to verify the checksum.
> >
>
> Now, I'm not saying this attack is particularly practical - it would
> generate a fair number of checkpoint failures before getting the first
> collision. So it'd trigger quite a few alerts, I guess.

You probably mean "checksum failures". I agree. And even if the checksum
passed the verification, page or tuple headers would probably be incorrect and
cause other errors.

> >> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
> >> a cryptographic hash algorithm. Now, maybe we don't want authenticated
> >> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).
> >
> >I'm also not sure if we should try to guarantee data authenticity /
> >integrity. As someone already mentioned elsewhere, page MAC does not help if
> >the whole page is replaced. (An extreme case is that old filesystem snapshot
> >containing the whole data directory is restored, although that will probably
> >make the database crash soon.)
> >
> >We can guarantee integrity and authenticity of backup, but that's a separate
> >feature: someone may need this although it's o.k. for him to run the cluster
> >unencrypted.
> >
>
> Yes, I do agree with that. I think attempts to guarantee data authenticity
> and/or integrity at the page level is mostly futile (replay attacks are an
> example of why). IMHO we should consider that to be outside the threat
> model TDE is expected to address.

When writing my previous email I forgot that, besides improving data
integrity, the authenticated encryption also tries to detect an attempt to get
encryption key via "chosen-ciphertext attack (CCA)". The fact that pages are
encrypted / decrypted independent from each other should not be a problem
here. We just need to consider if this kind of CCA is the threat we try to
protect against.

> IMO a better way to handle authenticity/integrity would be based on WAL,
> which is essentially an authoritative log of operations. We should be able
> to parse WAL, deduce expected state (min LSN, checksums) for each page,
> and validate the cluster state based on that.

ok. A replica that was cloned from the master before any corruption could have
happened can be used for such checks. But that should be done by an external
tool rather than by PG core.

> I still think having to decrypt the page in order to verify a checksum
> (because the header is part of the encrypted page, and is computed from
> the plaintext version) is not great.

Should we forbid the checksums if the cluster is encrypted? Even if the
checksum is encrypted, I think it can still help to detect I/O corruption: if
the encrypted data is corrupted, then the checksum verification should fail
after decryption anyway.

--
Antonin Houska
Web: https://www.cybertec-postgresql.com

On Fri, Jul 19, 2019 at 04:02:19PM +0200, Antonin Houska wrote:
>Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>
>> On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote:
>> >Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>> >
>> >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote:
>> >> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote:
>> >> >> One extra thing we should consider is authenticated encryption. We can't
>> >> >> just encrypt the pages (no matter which AES mode is used - XTS/CBC/...),
>> >> >> as that does not provide integrity protection (i.e. can't detect when
>> >> >> the ciphertext was corrupted due to disk failure or intentionally). And
>> >> >> we can't quite rely on checksums, because that checksums the plaintext
>> >> >> and is stored encrypted.
>> >> >
>> >> >Uh, if someone modifies a few bytes of the page, we will decrypt it, but
>> >> >the checksum (per-page or WAL) will not match our decrypted output.  How
>> >> >would they make it match the checksum without already knowing the key.
>> >> >I read [1] but could not see that explained.
>> >> >
>> >>
>> >> Our checksum is only 16 bits, so perhaps one way would be to just
>> >> generate 64k of randomly modified pages and hope one of them happens to
>> >> hit the right checksum value. Not sure how practical such attack is, but
>> >> it does require just filesystem access.
>> >
>> >I don't think you can easily generate 64k of different checksums this way. If
>> >the data is random, I suppose that each set of 2^(128 - 16) blocks will
>> >contain the the same checksum after decryption. Thus even you generate 64k of
>> >different ciphertext blocks that contain the checksum, some (many?)  checksums
>> >will be duplicate. Unfortunately the math to describe this problem does not
>> >seem to be trivial.
>> >
>>
>> I'm not sure what's your point, or why you care about the 128 bits, but I
>> don't think the math is very complicated (and it's exactly the same with
>> or without encryption). The probability of checksum collision for randomly
>> modified page is 1/64k, so p=~0.00153%. So probability of *not* getting a
>> collision is (1-p)=99.9985%. So with N pages, the probability of no
>> collisions is pow((1-p),N) which behaves like this:
>>
>>      N     pow((1-p),N)
>>    --------------------
>>    10000           85%
>>    20000           73%
>>    30000           63%
>>    46000           49%
>>    200000           4%
>>
>> So with 1.6GB relation you have about 96% chance of a checksum collision.
>
>I thought your attack proposal was to find valid (encrypted) checksum for a
>given encrypted page. Instead it seems that you were only trying to say that
>it's not too hard to generate page with a valid checksum in general. Thus the
>attacker can try to modify the ciphertext again and again in a way that is not
>quite random, but the chance to pass the checksum verification may still be
>relatively high.
>
>> >Also note that if you try to generate ciphertext, decryption of which will
>> >result in particular value of checksum, you can hardly control the other 14
>> >bytes of the block, which in turn are used to verify the checksum.
>> >
>>
>> Now, I'm not saying this attack is particularly practical - it would
>> generate a fair number of checkpoint failures before getting the first
>> collision. So it'd trigger quite a few alerts, I guess.
>
>You probably mean "checksum failures". I agree. And even if the checksum
>passed the verification, page or tuple headers would probably be incorrect and
>cause other errors.
>
>> >> FWIW our CRC algorithm is not quite HMAC, because it's neither keyed nor
>> >> a cryptographic hash algorithm. Now, maybe we don't want authenticated
>> >> encryption (e.g. XTS is not authenticated, unlike GCM/CCM).
>> >
>> >I'm also not sure if we should try to guarantee data authenticity /
>> >integrity. As someone already mentioned elsewhere, page MAC does not help if
>> >the whole page is replaced. (An extreme case is that old filesystem snapshot
>> >containing the whole data directory is restored, although that will probably
>> >make the database crash soon.)
>> >
>> >We can guarantee integrity and authenticity of backup, but that's a separate
>> >feature: someone may need this although it's o.k. for him to run the cluster
>> >unencrypted.
>> >
>>
>> Yes, I do agree with that. I think attempts to guarantee data authenticity
>> and/or integrity at the page level is mostly futile (replay attacks are an
>> example of why). IMHO we should consider that to be outside the threat
>> model TDE is expected to address.
>
>When writing my previous email I forgot that, besides improving data
>integrity, the authenticated encryption also tries to detect an attempt to get
>encryption key via "chosen-ciphertext attack (CCA)". The fact that pages are
>encrypted / decrypted independent from each other should not be a problem
>here. We just need to consider if this kind of CCA is the threat we try to
>protect against.
>
>> IMO a better way to handle authenticity/integrity would be based on WAL,
>> which is essentially an authoritative log of operations. We should be able
>> to parse WAL, deduce expected state (min LSN, checksums) for each page,
>> and validate the cluster state based on that.
>
>ok. A replica that was cloned from the master before any corruption could have
>happened can be used for such checks. But that should be done by an external
>tool rather than by PG core.
>
>> I still think having to decrypt the page in order to verify a checksum
>> (because the header is part of the encrypted page, and is computed from
>> the plaintext version) is not great.
>
>Should we forbid the checksums if the cluster is encrypted? Even if the
>checksum is encrypted, I think it can still help to detect I/O corruption: if
>the encrypted data is corrupted, then the checksum verification should fail
>after decryption anyway.
>

Forbid checksums? I don't see how that could be acceptable. We either have
to accept the limitations of the current design (having to decrypt
everything before checking the checksums) or change the design.

I personally think we should do the latter - not just because of this
"decrypt-then-verify" issue, but consider how much work we've done to
allow enabling checksums on-line (it's still not there, but it's likely
doable in PG13). How are we going to do that with encryption? ISTM we
should design it so that we can enable encryption on-line too - maybe not
in v1, but it should be possible. So how are we going to do that? With
checksums it's (fairly) easy because we can "not verify" the page before
we know all pages have a checksum, but with encryption that's not
possible. And if the whole page is encrypted, then what?

Of course, maybe we don't need such capability for the use-case we're
trying to solve with encryption. I can imagine that someone is running a
large system, has issues with data corruption, and decides to enable
checksums to remedy that. Maybe there's no such scenario in the privacy
case? But we can probably come up with scenarios where a new company
policy forces people to enable encryption on all systems, or something
like that.

That being said, I don't know how to solve this, but it seems to me that
any system where we can't easily decide whether a page is encrypted or not
(because everything including the page header) is encrypted has this
exact issue. Maybe we could keep some part of the header unencrypted
(likely an information leak, does not solve decrypt-then-verify). Or maybe
we need to store some additional information on each page (which breaks
on-disk format).

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Jul 20, 2019 at 1:30 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Forbid checksums? I don't see how that could be acceptable. We either have
> to accept the limitations of the current design (having to decrypt
> everything before checking the checksums) or change the design.
>
> I personally think we should do the latter - not just because of this
> "decrypt-then-verify" issue, but consider how much work we've done to
> allow enabling checksums on-line (it's still not there, but it's likely
> doable in PG13). How are we going to do that with encryption? ISTM we
> should design it so that we can enable encryption on-line too - maybe not
> in v1, but it should be possible. So how are we going to do that? With
> checksums it's (fairly) easy because we can "not verify" the page before
> we know all pages have a checksum, but with encryption that's not
> possible. And if the whole page is encrypted, then what?
>
> Of course, maybe we don't need such capability for the use-case we're
> trying to solve with encryption. I can imagine that someone is running a
> large system, has issues with data corruption, and decides to enable
> checksums to remedy that. Maybe there's no such scenario in the privacy
> case? But we can probably come up with scenarios where a new company
> policy forces people to enable encryption on all systems, or something
> like that.
>
> That being said, I don't know how to solve this, but it seems to me that
> any system where we can't easily decide whether a page is encrypted or not
> (because everything including the page header) is encrypted has this
> exact issue. Maybe we could keep some part of the header unencrypted
> (likely an information leak, does not solve decrypt-then-verify). Or maybe
> we need to store some additional information on each page (which breaks
> on-disk format).

How about storing the CRC of the encrypted pages? It would not leak
any additional information and serves the same purpose as a
non-encrypted one, namely I/O corruption detection. I took a look at
pg_checksum and besides checking for empty pages, the checksum
validation path does not interpret any other fields to calculate the
checksum. I think even the offline checksum enabling path looks like
it may work out of the box. Checksums of encrypted data are not a
replacement for a MAC and this would allow that validation to run
without any knowledge of keys.

Related, I think CTR mode should be considered for pages. It has
performance advantages at 8K data sizes, but even better, allows for
arbitrary bytes of the cipher text to be replaced. For example, after
encrypting a block you can replace the two checksum bytes with the CRC
of the cipher text v.s. CBC mode where that would cause corruption to
cascade forward. Same could be used for leaving things like
pd_pagesize_version in plaintext at its current offset. For anything
deemed non-sensitive, having it readable without having to decrypt the
page is useful.

It does not have to be full bytes either. CTR mode operates as a
stream of bits so its possible to replace nibbles or even individual
bits. It can be something as small one bit for an "is_encrypted" flag
or a handful of bits used to infer a derived key. For example, with
2-bits you could have 00 represent unencrypted, 01/10 represent
old/new key, and 11 be future use. Something like that could
facilitate online key rotation.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

On Mon, Jul 15, 2019 at 07:39:20PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-15, Bruce Momjian wrote:
> 
> > My point is that doing encryption of only some data might actually make
> > the system slower due to the lookups, so I think we need to implement
> > all-cluster encryption and then see what the overhead is, and if there
> > are use-cases for not encrypting only some data.
> 
> We can keep the keys in the relcache.  It doesn't have to be slow.  It
> is certainly slower to have to encrypt *all* data, which can be
> massively larger than the sensitive portion of the database.
> 
> If we need the keys for offline operation (where relcache is not
> reachable), we can keep pointers to the key files in the filesystem --
> for example for an encrypted table we would keep a new file, say
> <relfilenode>.key, which could be a symlink to the encrypted key file.
> The tool already has access to the key data, but the symlink lets it
> know *which* key to use; random onlookers cannot get the key data
> because the file is encrypted with the master key.
> 
> Any table without the key file is assumed to be unencrypted.

The relcache and symlinks is an interesting idea.  Are we still
encrypting all of WAL?  If so, the savings is only on heap/index file
writes, and I just don't know much of a benefit skipping encryption will
be --- we can test it later.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 20, 2019 at 03:39:25PM -0400, Sehrope Sarkuni wrote:
> How about storing the CRC of the encrypted pages? It would not leak
> any additional information and serves the same purpose as a
> non-encrypted one, namely I/O corruption detection. I took a look at
> pg_checksum and besides checking for empty pages, the checksum
> validation path does not interpret any other fields to calculate the
> checksum. I think even the offline checksum enabling path looks like
> it may work out of the box. Checksums of encrypted data are not a
> replacement for a MAC and this would allow that validation to run
> without any knowledge of keys.
> 
> Related, I think CTR mode should be considered for pages. It has
> performance advantages at 8K data sizes, but even better, allows for
> arbitrary bytes of the cipher text to be replaced. For example, after
> encrypting a block you can replace the two checksum bytes with the CRC
> of the cipher text v.s. CBC mode where that would cause corruption to
> cascade forward. Same could be used for leaving things like
> pd_pagesize_version in plaintext at its current offset. For anything
> deemed non-sensitive, having it readable without having to decrypt the
> page is useful.

Yes, I did cover that here:

    https://www.postgresql.org/message-id/20190716002519.yyvgl7qi4ewl6pc2@momjian.us
    
    Yes, it would only work if the checksum was the last part of the page,
    or if we used CTR mode, where changing the source bits doesn't affect
    the later bits.  I am thinking crazy here, I know, but it seemed worth
    mentioning in case someone liked it.
    
    https://www.postgresql.org/message-id/20190715194239.iqq5jdj54ru32kmt@momjian.us
    
    If we want to go crazy, we could encrypt, assume zeros for the CRC,
    compute the MAC and put it in the place of the CRC is, but then tools
    that read CRC would see that as an error, so we don't want to go there.
    Yes, crazy.

I know this thread is long so you might have missed it.

I do think CTR mode is the way to go for the heap/index pages and the
WAL, and will reply to another email on that topic now.

> It does not have to be full bytes either. CTR mode operates as a
> stream of bits so its possible to replace nibbles or even individual
> bits. It can be something as small one bit for an "is_encrypted" flag
> or a handful of bits used to infer a derived key. For example, with
> 2-bits you could have 00 represent unencrypted, 01/10 represent
> old/new key, and 11 be future use. Something like that could
> facilitate online key rotation.

Yes, if we do all-cluster encryption, we can just consult pg_control,
but if we do per-table/index, that might be needed.  There is another
email suggesting symlink file to a key file could indicate an encrypted
table/index.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote:
> On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian <bruce@momjian.us> wrote:
> > then each row change gets its own LSN.  You are asking if an update that
> > just expires one row and adds it to a new page gets the same LSN.  I
> > don't know.
> 
> The following scripts can reproduce that different two pages have the same LSN.
> 
> =# create table test (a int);
> CREATE TABLE
> =# insert into test select generate_series(1, 226);
> INSERT 0 226
> =# update test set a = a where a = 1;
> UPDATE 1
> =# select lsn from page_header(get_raw_page('test', 0));
>     lsn
> -----------
>  0/1690488
> (1 row)
> 
> =# select lsn from page_header(get_raw_page('test', 1));
>     lsn
> -----------
>  0/1690488
> (1 row)
> 
> So I think it's better to use LSN and page number to create IV. If we
> modify different tables by single WAL we also would need OID or
> relfilenode but I don't think currently we have such operations.

OK, good to know, thanks.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote:
> I've re-considered the design of TDE feature based on the discussion
> so far. The one of the main open question is the granular of
> encryption objects: cluster encryption or more-granular-than-cluster
> encryption. The followings describe about the new TDE design when we
> choose table-level encryption or something-new-group-level encryption.
> 
> General
> ========
> We will use AES and support both AES-128 and AES-256. User can specify
> the new initdb option something like --aes-128 or --aes-256 to enable
> encryption and must specify --encryption-key-passphrase-command along
> with. (I guess we also require openssl library.) If these options are
> specified, we write the key length to the control file and derive the
> KEK and generate MDEK during initdb. wal_log_hints will be enabled
> automatically in encryption mode, like we do for checksum mode,

Agreed.  pg_control will store the none/AES128/AES256 indicator.

> Key Management
> ==============
> We will use 3-tier key architecture as Joe proposed.
> 
>   1. A master key encryption key (KEK): this is the ley supplied by the
>      database admin using something akin to ssl_passphrase_command
> 
>   2. A master data encryption key (MDEK): this is a generated key using a
>      cryptographically secure pseudo-random number generator. It is
>      encrypted using the KEK, probably with Key Wrap (KW):
>      or maybe better Key Wrap with Padding (KWP):
> 
>   3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
>       table specific keys.

What is the value of a per-table encryption key?  How is HKDF derived?
Are we still unclear if the 68GB limit is per encryption key or per
encryption key/IV combination?
 
>   3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
>       generate new keys when needed for WAL.
> 
> We store MDEK to the plain file (say global/pgkey) after encrypted
> with the KEK. I might want to store the hash of passphrase of the KEK
> in order to verify the correctness of the given passphrase. However we
> don't need to store TDEK and WDEK as we can derive them as needed. The
> key file can be read by both backend processes and front-end tools.

Yes, we need to verify the pass phrase.

> When postmaster startup, it reads the key file and decrypts MDEK and
> derive WDEK using key id for WDEK. WDEK is loaded to the key hash map
> (keyid -> key) on the shared memory. Also we derive TDEK as needed
> when reading tables or indexes and add it to the key hash map as well
> if not exists.
> 
> Buffer Encryption
> ==============
> We will use AES-CBC for buffer encryption. We will add key id (4byte)

I think we might want to use CTR for this, and will post after this.

> to after the pd_lsn(8byte) in PageHeaderData and we will not encrypt
> first 16 byte of each pages so the LSN and key id can be used. We can
> store an invalid key id to tell us that the table is not encrypted.
> There two benefits of storing key id to the page header: offline tools
> can get key id (and know the table is encrypted or not) and it's
> helpful for online rekey in the future.

I don't remember anyone suggesting different keys for different tables. 
How would this even be managed by the user?

> I've considered to store IV and key id to a new fork but I felt that
> it is complex because we will always need to have the fork on the
> shared buffer when any pages of its main fork is written to the disk.
> If almost buffers of the shared buffers are dirtied and theirs new
> forks are not  loaded to the shared buffer, we might need to load the
> new fork and write the page to the disk and then evict some pages,
> over and over.
> 
> We will use (page lsn, page number) to create a nonce. IVs are created
> by encrypting the nonce with its TDEK.

Agreed.

> WAL Encryption
> =============
> We will use AES-CTR for WAL encryption and encrypt each WAL pages with WDEK.
> 
> We will use WAL segment number to create a nonce. Similar to buffer
> encryption, IVs are created using by the nonce and WDEK.

Yes.  If there is concern about collision of table/index and WAL IVs, we
can add a constant to the two uses, as Joe Conway mentioned.

> If we want to support enabling or disabling encryption after initdb we
> might want to have key id in the WAL page header.
> 
> Front-end Tool Support
> ==================
> We will add --encryption-key-passphrase-command option to the
> front-end tools that read database files or WAL segment files directly.
> They can get KEK via --encryption-key-passphrase-command and get MDEK
> by reading the key file. Also they can know the key length by checking
> the control file. Since they can derive TDEK using by key id stored in
> the page header they can decrypt database files. Similarly, they also
> can decrypt WAL as they can know the key id of WDEK.
>
> Master Key Rotation
> ================
> We will support new command-line tool that rotates the master key
> offline. It accepts --old-encryption-key-passphrase-command option and
> --new-encryption-key-passphrase-command to get old KEK and new KEK
> respectively. It decrypt MDEK with the old key and encrypt it with
> the new key.

That handles changing the passphrase, but what about rotating the
encryption key?  Don't we want to support that, at least in offline
mode?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 25, 2019 at 01:18:44PM -0400, Bruce Momjian wrote:
> > Key Management
> > ==============
> > We will use 3-tier key architecture as Joe proposed.
> > 
> >   1. A master key encryption key (KEK): this is the ley supplied by the
> >      database admin using something akin to ssl_passphrase_command
> > 
> >   2. A master data encryption key (MDEK): this is a generated key using a
> >      cryptographically secure pseudo-random number generator. It is
> >      encrypted using the KEK, probably with Key Wrap (KW):
> >      or maybe better Key Wrap with Padding (KWP):
> > 
> >   3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
> >       table specific keys.
> 
> What is the value of a per-table encryption key?  How is HKDF derived?
> Are we still unclear if the 68GB limit is per encryption key or per
> encryption key/IV combination?

Oh, I see you got this from Joe Conway's email.  Let me reply to that
now.  (I am obviously having problems keeping this thread in my head as
well.)

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote:
> > I've re-considered the design of TDE feature based on the discussion
> > so far. The one of the main open question is the granular of
> > encryption objects: cluster encryption or more-granular-than-cluster
> > encryption. The followings describe about the new TDE design when we
> > choose table-level encryption or something-new-group-level encryption.
> >
> > General
> > ========
> > We will use AES and support both AES-128 and AES-256. User can specify
> > the new initdb option something like --aes-128 or --aes-256 to enable
> > encryption and must specify --encryption-key-passphrase-command along
> > with. (I guess we also require openssl library.) If these options are
> > specified, we write the key length to the control file and derive the
> > KEK and generate MDEK during initdb. wal_log_hints will be enabled
> > automatically in encryption mode, like we do for checksum mode,
>
> Agreed.  pg_control will store the none/AES128/AES256 indicator.
>
> > Key Management
> > ==============
> > We will use 3-tier key architecture as Joe proposed.
> >
> >   1. A master key encryption key (KEK): this is the ley supplied by the
> >      database admin using something akin to ssl_passphrase_command
> >
> >   2. A master data encryption key (MDEK): this is a generated key using a
> >      cryptographically secure pseudo-random number generator. It is
> >      encrypted using the KEK, probably with Key Wrap (KW):
> >      or maybe better Key Wrap with Padding (KWP):
> >
> >   3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
> >       table specific keys.
>
> What is the value of a per-table encryption key?  How is HKDF derived?

Per-table encryption key is derived from MDEK with salt and its OID as
info. I think we can store salts for each encryption keys into the
separate file so that off-line tool also can read it.

> Are we still unclear if the 68GB limit is per encryption key or per
> encryption key/IV combination?

I think that 68GB refers to key+IV but I'll research that.

>
> >   3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
> >       generate new keys when needed for WAL.
> >
> > We store MDEK to the plain file (say global/pgkey) after encrypted
> > with the KEK. I might want to store the hash of passphrase of the KEK
> > in order to verify the correctness of the given passphrase. However we
> > don't need to store TDEK and WDEK as we can derive them as needed. The
> > key file can be read by both backend processes and front-end tools.
>
> Yes, we need to verify the pass phrase.
>
> > When postmaster startup, it reads the key file and decrypts MDEK and
> > derive WDEK using key id for WDEK. WDEK is loaded to the key hash map
> > (keyid -> key) on the shared memory. Also we derive TDEK as needed
> > when reading tables or indexes and add it to the key hash map as well
> > if not exists.
> >
> > Buffer Encryption
> > ==============
> > We will use AES-CBC for buffer encryption. We will add key id (4byte)
>
> I think we might want to use CTR for this, and will post after this.
>
> > to after the pd_lsn(8byte) in PageHeaderData and we will not encrypt
> > first 16 byte of each pages so the LSN and key id can be used. We can
> > store an invalid key id to tell us that the table is not encrypted.
> > There two benefits of storing key id to the page header: offline tools
> > can get key id (and know the table is encrypted or not) and it's
> > helpful for online rekey in the future.
>
> I don't remember anyone suggesting different keys for different tables.
> How would this even be managed by the user?

I think it's still unclear whether we implement one key for whole
database cluster or different keys for different table as the first
version. I'm evaluating the performance overhead of the latter that
you concerned and will share it.

I prefer tablespace-level or something-new-group-level than
table-level but if we choose the latter we can create a new group of
tables that are encrypted with the same key. That is user create a
group and then associate tables to that group. Tablespace-level is
implemented in the patch I submitted before. Or it's just idea but
another idea could be to allow users to create encryption key object
first and then specify which tables are encrypted with which
encryption key in DDL. For example, user creates an encryption keys
with name by SQL function and creates an encrypted table by CREATE
TABLE ... WITH (encryption_key = 'mykey');.

>
> > I've considered to store IV and key id to a new fork but I felt that
> > it is complex because we will always need to have the fork on the
> > shared buffer when any pages of its main fork is written to the disk.
> > If almost buffers of the shared buffers are dirtied and theirs new
> > forks are not  loaded to the shared buffer, we might need to load the
> > new fork and write the page to the disk and then evict some pages,
> > over and over.
> >
> > We will use (page lsn, page number) to create a nonce. IVs are created
> > by encrypting the nonce with its TDEK.
>
> Agreed.
>
> > WAL Encryption
> > =============
> > We will use AES-CTR for WAL encryption and encrypt each WAL pages with WDEK.
> >
> > We will use WAL segment number to create a nonce. Similar to buffer
> > encryption, IVs are created using by the nonce and WDEK.
>
> Yes.  If there is concern about collision of table/index and WAL IVs, we
> can add a constant to the two uses, as Joe Conway mentioned.
>
> > If we want to support enabling or disabling encryption after initdb we
> > might want to have key id in the WAL page header.
> >
> > Front-end Tool Support
> > ==================
> > We will add --encryption-key-passphrase-command option to the
> > front-end tools that read database files or WAL segment files directly.
> > They can get KEK via --encryption-key-passphrase-command and get MDEK
> > by reading the key file. Also they can know the key length by checking
> > the control file. Since they can derive TDEK using by key id stored in
> > the page header they can decrypt database files. Similarly, they also
> > can decrypt WAL as they can know the key id of WDEK.
> >
> > Master Key Rotation
> > ================
> > We will support new command-line tool that rotates the master key
> > offline. It accepts --old-encryption-key-passphrase-command option and
> > --new-encryption-key-passphrase-command to get old KEK and new KEK
> > respectively. It decrypt MDEK with the old key and encrypt it with
> > the new key.
>
> That handles changing the passphrase, but what about rotating the
> encryption key?  Don't we want to support that, at least in offline
> mode?

Yeah, supporting rotating the encryption key is a good idea. Agreed.

After more thoughts, it's a just idea but I wonder if the first
implementation step of TDE for v13 could be key management module.
That is, (in 3-tier case) PostgreSQL gets KEK by passphrase command or
directly, and creates MDEK. User can create an encryption key with
name using by SQL function, and the key manager derives DEK and store
its salt to the disk. Also we have an internal interface to get an
encryption key.

The good point is not only to develop incrementally but also that if
PostgreSQL is able to manage (symmetric) encryption keys inside
database cluster and has interfaces to get and add keys, pgcrypt also
will be able to use it. That way, we will provide column-level TDE
first by combination of pgcrypt, triggers and views while keeping
encryption keys truly secret. After that we can add other level TDE
using the key management module. We would then be able to focus on how
to encrypt buffer and WAL.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote:
> In my email I linked the wrong page for [2]. The correct one is here:
> [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html
> 
> Following that, I think we could end up with three tiers:
> 
> 1. A master key encryption key (KEK): this is the ley supplied by the
>    database admin using something akin to ssl_passphrase_command
> 
> 2. A master data encryption key (MDEK): this is a generated key using a
>    cryptographically secure pseudo-random number generator. It is
>    encrypted using the KEK, probably with Key Wrap (KW):
>    or maybe better Key Wrap with Padding (KWP):
> 
> 3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
>     table specific keys.
> 
> 3b. WAL data encryption keys (WDEK):  Similarly use MDEK and a HKDF to
>     generate new keys when needed for WAL (based on the other info we
>     need to change WAL keys every 68 GB unless I read that wrong).
> 
> I believe that would allows us to have multiple keys but they are
> derived securely from the one DEK using available info similar to the
> way we intend to use LSN to derive the IVs -- perhaps table.oid for
> tables and something else for WAL.
> 
> We also need to figure out how/when to generate new WDEK. Maybe every
> checkpoint, also meaning we would have to force a checkpoint every 68GB?

Masahiko Sawada copied this section as a desired direction, so I want to
drill down into it.  I think we have five possible approaches for level
3 listed above.

The simplest approach would be to say that the LSN/page-number and WAL
segment-number used as IV will not collide and we can just use them
directly.

The second approach is to say they will collide and that we need to mix
a constant into the IV for tables/indexes and a different one for WAL. 
In a way I would like to mix the pg_controldata Database system
Identifier into there too, but I am unclear on the value and complexity
involved.

A third approach would be to say that we will have duplicate LSNs
between a table and its index?  Maybe we need three constants, one for
heap, one for indexes, and one for WAL.

A fourth approach would be to say we will have duplicate LSNs on
different heap files, or index files.  We would then modify pg_upgrade to
preserve relfilenode and use that.  (I don't think pg_class.oid is
visible during recovery, and it certainly isn't visible in offline
mode.)

However, we need to be clear that adding relfilenode is only helping to
avoid tables/indexes with the same LSN pages.  It doesn't address the
68GB limit since our tables can be larger than that.

A fifth approach would be to decide that 68GB is the limit for a single
key (not single key/IV combo).  If that is the case we need a different
key for each 68GB of a file, and because we break files into 1GB chunks,
we would just use a different key for each chunk, and I guess store the
keys in the file system, encrypted with the master key.

My big point is that we need to decide where the IV collisions will
happen, and what our encryption limit per key (not per key/IV
combination) is.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 26, 2019 at 02:54:04AM +0900, Masahiko Sawada wrote:
> On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian <bruce@momjian.us> wrote:
> >
> > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote:
> > > I've re-considered the design of TDE feature based on the discussion
> > > so far. The one of the main open question is the granular of
> > > encryption objects: cluster encryption or more-granular-than-cluster
> > > encryption. The followings describe about the new TDE design when we
> > > choose table-level encryption or something-new-group-level encryption.
> > >
> > > General
> > > ========
> > > We will use AES and support both AES-128 and AES-256. User can specify
> > > the new initdb option something like --aes-128 or --aes-256 to enable
> > > encryption and must specify --encryption-key-passphrase-command along
> > > with. (I guess we also require openssl library.) If these options are
> > > specified, we write the key length to the control file and derive the
> > > KEK and generate MDEK during initdb. wal_log_hints will be enabled
> > > automatically in encryption mode, like we do for checksum mode,
> >
> > Agreed.  pg_control will store the none/AES128/AES256 indicator.
> >
> > > Key Management
> > > ==============
> > > We will use 3-tier key architecture as Joe proposed.
> > >
> > >   1. A master key encryption key (KEK): this is the ley supplied by the
> > >      database admin using something akin to ssl_passphrase_command
> > >
> > >   2. A master data encryption key (MDEK): this is a generated key using a
> > >      cryptographically secure pseudo-random number generator. It is
> > >      encrypted using the KEK, probably with Key Wrap (KW):
> > >      or maybe better Key Wrap with Padding (KWP):
> > >
> > >   3a. Per table data encryption keys (TDEK): use MDEK and HKDF to generate
> > >       table specific keys.
> >
> > What is the value of a per-table encryption key?  How is HKDF derived?
> 
> Per-table encryption key is derived from MDEK with salt and its OID as
> info. I think we can store salts for each encryption keys into the
> separate file so that off-line tool also can read it.

Thanks. I just sent an email with five possible options for this.  I
think relfilenode will be fine --- I am not sure what salt adds to this.

> > Are we still unclear if the 68GB limit is per encryption key or per
> > encryption key/IV combination?
> 
> I think that 68GB refers to key+IV but I'll research that.

Yes, please.  I think we need a definite answer on that question, which
you will see in my later email.

> > I don't remember anyone suggesting different keys for different tables.
> > How would this even be managed by the user?
> 
> I think it's still unclear whether we implement one key for whole
> database cluster or different keys for different table as the first
> version. I'm evaluating the performance overhead of the latter that
> you concerned and will share it.

I am not worried about the performance of this --- if it not secure with
a single key, it is useless, so we have to do it.  If the single key is
secure, I don't think multiple keys helps us.  I think we already
decided that the keys always have to be online for crash recovery, so we
can't allow users to control their keys anyway.

> I prefer tablespace-level or something-new-group-level than
> table-level but if we choose the latter we can create a new group of
> tables that are encrypted with the same key. That is user create a
> group and then associate tables to that group. Tablespace-level is
> implemented in the patch I submitted before. Or it's just idea but
> another idea could be to allow users to create encryption key object
> first and then specify which tables are encrypted with which
> encryption key in DDL. For example, user creates an encryption keys
> with name by SQL function and creates an encrypted table by CREATE
> TABLE ... WITH (encryption_key = 'mykey');.

That seems very complex so I think we need agreement to go in that
direction, and see what I said above about multiple keys.

> > That handles changing the passphrase, but what about rotating the
> > encryption key?  Don't we want to support that, at least in offline
> > mode?
> 
> Yeah, supporting rotating the encryption key is a good idea. Agreed.
> 
> After more thoughts, it's a just idea but I wonder if the first
> implementation step of TDE for v13 could be key management module.
> That is, (in 3-tier case) PostgreSQL gets KEK by passphrase command or
> directly, and creates MDEK. User can create an encryption key with
> name using by SQL function, and the key manager derives DEK and store
> its salt to the disk. Also we have an internal interface to get an
> encryption key.
> 
> The good point is not only to develop incrementally but also that if
> PostgreSQL is able to manage (symmetric) encryption keys inside
> database cluster and has interfaces to get and add keys, pgcrypt also
> will be able to use it. That way, we will provide column-level TDE
> first by combination of pgcrypt, triggers and views while keeping
> encryption keys truly secret. After that we can add other level TDE
> using the key management module. We would then be able to focus on how
> to encrypt buffer and WAL.

Uh, remember, all keys have to be online all the time, so what value
does this add?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Sat, Jul 20, 2019 at 07:30:30PM +0200, Tomas Vondra wrote:
> Forbid checksums? I don't see how that could be acceptable. We either have
> to accept the limitations of the current design (having to decrypt
> everything before checking the checksums) or change the design.

Yes, checksums certainly have to work.

> I personally think we should do the latter - not just because of this
> "decrypt-then-verify" issue, but consider how much work we've done to
> allow enabling checksums on-line (it's still not there, but it's likely
> doable in PG13). How are we going to do that with encryption? ISTM we
> should design it so that we can enable encryption on-line too - maybe not
> in v1, but it should be possible. So how are we going to do that? With
> checksums it's (fairly) easy because we can "not verify" the page before
> we know all pages have a checksum, but with encryption that's not
> possible. And if the whole page is encrypted, then what?

Well, I assumed we would start with a command-line offline tool to
add/remove encryption, and I assumed the command-line tool pg_checksums
would use the same code to decrypt the page to add/remove checksums and
rewrite it.  I don't think we will ever allow add/remove encryption in
online mode.

Does that help?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Fri, Jul 19, 2019 at 01:59:41PM +0200, Tomas Vondra wrote:
> On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote:
> > We can guarantee integrity and authenticity of backup, but that's a separate
> > feature: someone may need this although it's o.k. for him to run the cluster
> > unencrypted.

> Yes, I do agree with that. I think attempts to guarantee data authenticity
> and/or integrity at the page level is mostly futile (replay attacks are an
> example of why). IMHO we should consider that to be outside the threat
> model TDE is expected to address.

Yes, I think we can say that checksums _help_ detect unauthorized
database changes, and usually detects database corruption, but it isn't
a fully secure solution.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote:
> The second approach is to say they will collide and that we need to mix
> a constant into the IV for tables/indexes and a different one for WAL. 
> In a way I would like to mix the pg_controldata Database system
> Identifier into there too, but I am unclear on the value and complexity
> involved.
> 
> A third approach would be to say that we will have duplicate LSNs
> between a table and its index?  Maybe we need three constants, one for
> heap, one for indexes, and one for WAL.
> 
> A fourth approach would be to say we will have duplicate LSNs on
> different heap files, or index files.  We would then modify pg_upgrade to
> preserve relfilenode and use that.  (I don't think pg_class.oid is
> visible during recovery, and it certainly isn't visible in offline
> mode.)
> 
> However, we need to be clear that adding relfilenode is only helping to
> avoid tables/indexes with the same LSN pages.  It doesn't address the
> 68GB limit since our tables can be larger than that.
> 
> A fifth approach would be to decide that 68GB is the limit for a single
> key (not single key/IV combo).  If that is the case we need a different
> key for each 68GB of a file, and because we break files into 1GB chunks,
> we would just use a different key for each chunk, and I guess store the
> keys in the file system, encrypted with the master key.
> 
> My big point is that we need to decide where the IV collisions will
> happen, and what our encryption limit per key (not per key/IV
> combination) is.

After talking to Joe Conway, I just want to mention that if we decide
that the LSN is unique among heap and index, or among heap or index, we
will need to make sure future WAL records retain this uniqueness.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> After talking to Joe Conway, I just want to mention that if we decide
> that the LSN is unique among heap and index, or among heap or index, we
> will need to make sure future WAL records retain this uniqueness.

One thing comes to mind regarding this and I'll admit that I don't quite
remember exactly off-hand but I also don't want to not mention it now
and forget to later.

What about pg_upgrade?

Thanks,

Stephen

On 2019-Jul-15, Bruce Momjian wrote:

> Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> the checksum (per-page or WAL) will not match our decrypted output.  How
> would they make it match the checksum without already knowing the key. 
> I read [1] but could not see that explained.
> 
> This post discussed it:
> 
>     https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac

I find all the discussion downthread from this post pretty confusing.
Why are we encrypting the page header in the first place?  It seems to
me that the encrypted area should cover only the line pointers and the
tuple data area; the page header needs to be unencrypted so that it can
be used at all: firstly because you need to obtain the LSN from it in
order to compute the IV, and secondly because the checksum must be
validated *before* decrypting (per Moxie Marlinspike's "cryptographic
doom" principle mentioned in a comment in the SE question).

I am not totally clear on whether the special space and the "page hole"
need to be encrypted.  I tend to think that they should *not* be
encrypted; in particular, encrypting a large area containing zeroes seem
a plentiful source of known cleartext, which seems a bad thing.  Special
space also seems to contain known cleartext; maybe not as much as the
page hole, but still seems better avoided.

Given this, it seems to me that we should first encrypt those two data
areas, and *then* compute the CRC on the complete page just like we do
today ... and the result is stored in an unencrypted area (the page
header) and so it doesn't affect the encryption.

The checksum we currently have is not cryptographically secure -- it's
not a crypto-strong signature.  If we want that, we need some further
protection.  Maybe for encrypted tables we replace our current checksum
with an cryptographically secure signature ...?  Pretty sure 16 bits are
insufficient for that, but I suppose we would just use a different page
header with room for a proper sig.

Am I misunderstanding something?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > After talking to Joe Conway, I just want to mention that if we decide
> > that the LSN is unique among heap and index, or among heap or index, we
> > will need to make sure future WAL records retain this uniqueness.
> 
> One thing comes to mind regarding this and I'll admit that I don't quite
> remember exactly off-hand but I also don't want to not mention it now
> and forget to later.
> 
> What about pg_upgrade?

So, we don't carry WAL from the old cluster to the new cluster, so if
the WAL is changed and had duplicates, it would only be new WAL records.
pg_upgrade seems immune to must of this, and that is by design. 
However, I am hesitant to change the heap/index page format for
encryption because if we add fields, old pages might not fit as
encrypted pages, and then you have to move rows around, and things
become _much_ more complicated.

I don't see any other pg_upgrade issues, unless someone else does.  Oh,
we will have to check pg_control for a matching encryption format.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote:
> > Greetings,
> >
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > After talking to Joe Conway, I just want to mention that if we decide
> > > that the LSN is unique among heap and index, or among heap or index, we
> > > will need to make sure future WAL records retain this uniqueness.
> >
> > One thing comes to mind regarding this and I'll admit that I don't quite
> > remember exactly off-hand but I also don't want to not mention it now
> > and forget to later.
> >
> > What about pg_upgrade?
>
> So, we don't carry WAL from the old cluster to the new cluster, so if
> the WAL is changed and had duplicates, it would only be new WAL records.

Right, we don't carry it forward- but what I couldn't remember is if
start from more-or-less LSN 0 or if pg_upgrade will arrange it such that
the new major version will start from LSN-of-old+1 (or whatever).  Seems
like it'd *have* to be the latter, but just thought of it and wanted to
make sure.

> pg_upgrade seems immune to must of this, and that is by design.
> However, I am hesitant to change the heap/index page format for
> encryption because if we add fields, old pages might not fit as
> encrypted pages, and then you have to move rows around, and things
> become _much_ more complicated.

Yeah, I'm afraid we are going to have a hard time making this work
without changing the page format for encrypted..  I don't know if that's
actually a *huge* issue like we've considered it to be in the past or
not, as making someone rewrite just the few sensitive tables in their
environment might not be that bad, and there's also logical replication
today..

> I don't see any other pg_upgrade issues, unless someone else does.  Oh,
> we will have to check pg_control for a matching encryption format.

Yes, certainly it'd need to be updated for at least that, when upgading
an encrypted cluster.

Thanks!

Stephen

On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote:
> On 2019-Jul-15, Bruce Momjian wrote:
> 
> > Uh, if someone modifies a few bytes of the page, we will decrypt it, but
> > the checksum (per-page or WAL) will not match our decrypted output.  How
> > would they make it match the checksum without already knowing the key. 
> > I read [1] but could not see that explained.
> > 
> > This post discussed it:
> > 
> >     https://crypto.stackexchange.com/questions/202/should-we-mac-then-encrypt-or-encrypt-then-mac
> 
> I find all the discussion downthread from this post pretty confusing.

Agreed.

> Why are we encrypting the page header in the first place?  It seems to
> me that the encrypted area should cover only the line pointers and the
> tuple data area; the page header needs to be unencrypted so that it can
> be used at all: firstly because you need to obtain the LSN from it in

Yes, the plan was to not encrypt the first 16 bytes so the LSN was visible.

> order to compute the IV, and secondly because the checksum must be
> validated *before* decrypting (per Moxie Marlinspike's "cryptographic
> doom" principle mentioned in a comment in the SE question).

Uh, I think we are still on the fence about writing the checksum _after_
encryption, but I think we are leaning against that, meaning online or
offline encryption must be able to decrypt the page.  Since we will
already need an offline tool to enable/remove encryption anyway, it
seems we can just reuse that code for pg_checksums.

I think we have three options with for CRC:

1.  compute CRC and then encrypt everything

2   encrypt and then CRC, and store the CRC unchanged

3.  encrypt and then CRC, and store the CRC encrypted

The only way offline tools can verify the CRC without access to the keys
is via #2, but #2 gives us _no_ detection of tampering.  I realize the
CRC tampering detection of #1 and #3 is not great, but it certainly has
some value.

> I am not totally clear on whether the special space and the "page hole"
> need to be encrypted.  I tend to think that they should *not* be
> encrypted; in particular, encrypting a large area containing zeroes seem
> a plentiful source of known cleartext, which seems a bad thing.  Special
> space also seems to contain known cleartext; maybe not as much as the
> page hole, but still seems better avoided.

Uh, there are no known attacks on AES with known plain-text, e.g., SSL
uses AES, so I think we are good with encrypting everything after the
first 16 bytes.

> Given this, it seems to me that we should first encrypt those two data
> areas, and *then* compute the CRC on the complete page just like we do
> today ... and the result is stored in an unencrypted area (the page
> header) and so it doesn't affect the encryption.

Yes, that is a possibility.

> The checksum we currently have is not cryptographically secure -- it's
> not a crypto-strong signature.  If we want that, we need some further
> protection.  Maybe for encrypted tables we replace our current checksum
> with an cryptographically secure signature ...?  Pretty sure 16 bits are
> insufficient for that, but I suppose we would just use a different page
> header with room for a proper sig.

Yes, checksum is more for best-effort than fully secure, but replay of
pages makes a fullly secure solution hard anyway.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote:
> Greetings,
> 
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote:
> > > Greetings,
> > > 
> > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > After talking to Joe Conway, I just want to mention that if we decide
> > > > that the LSN is unique among heap and index, or among heap or index, we
> > > > will need to make sure future WAL records retain this uniqueness.
> > > 
> > > One thing comes to mind regarding this and I'll admit that I don't quite
> > > remember exactly off-hand but I also don't want to not mention it now
> > > and forget to later.
> > > 
> > > What about pg_upgrade?
> > 
> > So, we don't carry WAL from the old cluster to the new cluster, so if
> > the WAL is changed and had duplicates, it would only be new WAL records.
> 
> Right, we don't carry it forward- but what I couldn't remember is if
> start from more-or-less LSN 0 or if pg_upgrade will arrange it such that
> the new major version will start from LSN-of-old+1 (or whatever).  Seems
> like it'd *have* to be the latter, but just thought of it and wanted to
> make sure.

pg_upgrade uses pg_resetwal -l to set the next WAL segment file based on
the value in the old cluster:

    /* now reset the wal archives in the new cluster */
    prep_status("Resetting WAL archives");
    exec_prog(UTILITY_LOG_FILE, NULL, true, true,
    /* use timeline 1 to match controldata and no WAL history file */
-->           "\"%s/pg_resetwal\" -l 00000001%s \"%s\"", new_cluster.bindir,
              old_cluster.controldata.nextxlogfile + 8,
              new_cluster.pgdata);

> > pg_upgrade seems immune to must of this, and that is by design. 
> > However, I am hesitant to change the heap/index page format for
> > encryption because if we add fields, old pages might not fit as
> > encrypted pages, and then you have to move rows around, and things
> > become _much_ more complicated.
> 
> Yeah, I'm afraid we are going to have a hard time making this work
> without changing the page format for encrypted..  I don't know if that's
> actually a *huge* issue like we've considered it to be in the past or
> not, as making someone rewrite just the few sensitive tables in their
> environment might not be that bad, and there's also logical replication
> today..

It is hard to do that while the server is offline.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote:
> Hi,
> 
> Some more thoughts on CBC vs CTR modes. There are a number of
> advantages to using CTR mode for page encryption.
> 
> CTR encryption modes can be fully parallelized, whereas CBC can only
> parallelized for decryption. While both can use AES specific hardware
> such as AES-NI, CTR modes can go a step further and use vectorized
> instructions.
> 
> On an i7-8559U (with AES-NI) I get a 4x speed improvement for
> CTR-based modes vs CBC when run on 8K of data:
> 
> # openssl speed -evp ${cipher}
> type             16 bytes     64 bytes    256 bytes   1024 bytes
> 8192 bytes  16384 bytes
> aes-128-cbc    1024361.51k  1521249.60k  1562033.41k  1571663.87k
> 1574537.90k  1575512.75k
> aes-128-ctr     696866.85k  2214441.86k  4364903.85k  5896221.35k
> 6559735.81k  6619594.75k
> aes-128-gcm     642758.92k  1638619.09k  3212068.27k  5085193.22k
> 6366035.97k  6474006.53k
> aes-256-cbc     940906.25k  1114628.44k  1131255.13k  1138385.92k
> 1140258.13k  1143592.28k
> aes-256-ctr     582161.82k  1896409.32k  3216926.12k  4249708.20k
> 4680299.86k  4706375.00k
> aes-256-gcm     553513.89k  1532556.16k  2705510.57k  3931744.94k
> 4615812.44k  4673093.63k

I am back to this email now.  I think there is a strong case that we
should use CTR mode for both WAL and heap/index files because CTR mode
is faster.  CBC mode has the advantage of being more immune to IV
duplication, but I think the fact that the page format is similar enough
among pages means we don't gain a lot from that, and therefor IV
uniqueness must be closely honored anyway.

> For relation data where the encryption is going to be per page,
> there's flexibility in how the CTR nonce (IV + counter) is generated.
> With an 8K page, the counter need only go up to 512 for each page
> (8192-bytes per page / 16-bytes per AES-block). That would require
> 9-bits for the counter. Rounding that up to 16-bits allows for wider
> pages and it still uses only two bytes of the counter while ensuring
> that it'd be unique per AES-block. The remaining 14-bytes would be
> populated with some other data that is guaranteed unique per
> page-write to allow encryption via the same per-relation-file derived
> key. From what I gather, the LSN is a candidate though it'd have to be
> stored in plaintext for decryption.

Yes, LSN is 8 bytes, and the page number is 4 bytes.  That leaves four
bytes of the counter.

> What's important is that writing the two pages (either different
> locations or the same page back again) never reuses the same nonce
> with the same key. Using the same nonce with a different key is fine.
> 
> With any of these schemes the same inputs will generate the same
> outputs. With CTR mode for WAL this would be an issue if the same key
> and deterministic nonce (ex: LSN + offset) is reused in multiple
> places. That does not have to be the same cluster either. For example
> if two replicas are promoted from the same backup with the same master
> key, they would generate the same WAL CTR stream, reusing the
> key/nonce pair. Ditto for starting off with a master key and deriving
> per-relation keys in a cloned installation off some deterministic
> attribute such as oid.

I think we need to document that sharing keys among clusters (except
for identical replicas) is insecure.

We can add the "Database system identifier" into the IV, which would
avoid the problem of two clusters using the same key, but it wouldn't
avoid the problem with promoting two replicas to primaries because they
would have the same "Database system identifier" so I think it is just
simpler to say "don't do that".

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote:
> > > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > > After talking to Joe Conway, I just want to mention that if we decide
> > > > > that the LSN is unique among heap and index, or among heap or index, we
> > > > > will need to make sure future WAL records retain this uniqueness.
> > > >
> > > > One thing comes to mind regarding this and I'll admit that I don't quite
> > > > remember exactly off-hand but I also don't want to not mention it now
> > > > and forget to later.
> > > >
> > > > What about pg_upgrade?
> > >
> > > So, we don't carry WAL from the old cluster to the new cluster, so if
> > > the WAL is changed and had duplicates, it would only be new WAL records.
> >
> > Right, we don't carry it forward- but what I couldn't remember is if
> > start from more-or-less LSN 0 or if pg_upgrade will arrange it such that
> > the new major version will start from LSN-of-old+1 (or whatever).  Seems
> > like it'd *have* to be the latter, but just thought of it and wanted to
> > make sure.
>
> pg_upgrade uses pg_resetwal -l to set the next WAL segment file based on
> the value in the old cluster:
>
>     /* now reset the wal archives in the new cluster */
>     prep_status("Resetting WAL archives");
>     exec_prog(UTILITY_LOG_FILE, NULL, true, true,
>     /* use timeline 1 to match controldata and no WAL history file */
> -->           "\"%s/pg_resetwal\" -l 00000001%s \"%s\"", new_cluster.bindir,
>               old_cluster.controldata.nextxlogfile + 8,
>               new_cluster.pgdata);

Ah, right, ok, we reset the timeline but not the LSN, ok.

> > > pg_upgrade seems immune to must of this, and that is by design.
> > > However, I am hesitant to change the heap/index page format for
> > > encryption because if we add fields, old pages might not fit as
> > > encrypted pages, and then you have to move rows around, and things
> > > become _much_ more complicated.
> >
> > Yeah, I'm afraid we are going to have a hard time making this work
> > without changing the page format for encrypted..  I don't know if that's
> > actually a *huge* issue like we've considered it to be in the past or
> > not, as making someone rewrite just the few sensitive tables in their
> > environment might not be that bad, and there's also logical replication
> > today..
>
> It is hard to do that while the server is offline.

I don't see any reason to assume we must only support encrypting
individual tables when the server is offline, or that we have to support
any option which involves the server being offline when it comes to
doing encryption.

I'm not against supporting a "shut down the server and then encrypt
everything and then start it up" option, but I don't see any
particularly good reason to explicitly design the system with that
use-case in mind.

There seems to be a strong thrust on this thread to assume that a
database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
therefore we have to shut down the server to do it), but I don't get why
that's the case, particularly if we support any kind of mixed setup
where there's some data that's encrypted and some that isn't, and since
we're talking about using different keys for different things, it seems
to me that we almost certainly should be able to easily say "well,
there's no key for this, so just don't go through the decrypt/encrypt
routines".

I can see an argument for why we might need to go through a restart and
possibly use some off-line tool when a user decides they want, say, the
WAL, or CLOG, or the other control files, to be encrypted, or basic
encryption capability to be set up for the cluster (like generating the
master key and storing it or making some changes to the control file to
indicate that some things in this cluster has encrypted bits), but
saying we must have the server offline to support encrypted tables that
have a different page format is a bit like saying we need to take the
server offline to add a new tableam (like zheap) and that we have to use
some off-line utility while the server is down to convert a given table
from heapam to zheap, isn't it?

Thanks,

Stephen

On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote:
> > > > pg_upgrade seems immune to must of this, and that is by design. 
> > > > However, I am hesitant to change the heap/index page format for
> > > > encryption because if we add fields, old pages might not fit as
> > > > encrypted pages, and then you have to move rows around, and things
> > > > become _much_ more complicated.
> > > 
> > > Yeah, I'm afraid we are going to have a hard time making this work
> > > without changing the page format for encrypted..  I don't know if that's
> > > actually a *huge* issue like we've considered it to be in the past or
> > > not, as making someone rewrite just the few sensitive tables in their
> > > environment might not be that bad, and there's also logical replication
> > > today..
> > 
> > It is hard to do that while the server is offline.
> 
> I don't see any reason to assume we must only support encrypting
> individual tables when the server is offline, or that we have to support
> any option which involves the server being offline when it comes to
> doing encryption.
> 
> I'm not against supporting a "shut down the server and then encrypt
> everything and then start it up" option, but I don't see any
> particularly good reason to explicitly design the system with that
> use-case in mind.

You are right that we can allow it online, but we haven't been
discussing these cases since it is easy to do this because we have
access to the keys.  I do think whatever code we use for checksum online
changes will be used for encryption online changes.  We would need a
per-page bit to indicate encryption, hopefully in the first 16 bytes.

> There seems to be a strong thrust on this thread to assume that a
> database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
> therefore we have to shut down the server to do it), but I don't get why
> that's the case, particularly if we support any kind of mixed setup
> where there's some data that's encrypted and some that isn't, and since
> we're talking about using different keys for different things, it seems
> to me that we almost certainly should be able to easily say "well,
> there's no key for this, so just don't go through the decrypt/encrypt
> routines".

No, we can't easily do different keys for different things since all the
keys have to be online for crash recovery, so there isn't much value to
having different keys since they always have to be online.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote:
> > > > > pg_upgrade seems immune to must of this, and that is by design.
> > > > > However, I am hesitant to change the heap/index page format for
> > > > > encryption because if we add fields, old pages might not fit as
> > > > > encrypted pages, and then you have to move rows around, and things
> > > > > become _much_ more complicated.
> > > >
> > > > Yeah, I'm afraid we are going to have a hard time making this work
> > > > without changing the page format for encrypted..  I don't know if that's
> > > > actually a *huge* issue like we've considered it to be in the past or
> > > > not, as making someone rewrite just the few sensitive tables in their
> > > > environment might not be that bad, and there's also logical replication
> > > > today..
> > >
> > > It is hard to do that while the server is offline.
> >
> > I don't see any reason to assume we must only support encrypting
> > individual tables when the server is offline, or that we have to support
> > any option which involves the server being offline when it comes to
> > doing encryption.
> >
> > I'm not against supporting a "shut down the server and then encrypt
> > everything and then start it up" option, but I don't see any
> > particularly good reason to explicitly design the system with that
> > use-case in mind.
>
> You are right that we can allow it online, but we haven't been
> discussing these cases since it is easy to do this because we have
> access to the keys.  I do think whatever code we use for checksum online
> changes will be used for encryption online changes.  We would need a
> per-page bit to indicate encryption, hopefully in the first 16 bytes.

Arranging to have an individual table move from being plain to
encrypted is something that would be nice to support in an online and
non-blocking manner, but *that* is a bunch of additional work that we
don't need to do today as opposed to being something that's just part of
the initial design.  Sure, it might use functions/capabilities that
pg_checksums also use, but I don't know that we need to think about the
code sharing there being much more than that, just that those
capabilities should be built out in a way that they can be used for
multiple things (and based on what I saw, that looks like it's exactly
how that code was being written already).

> > There seems to be a strong thrust on this thread to assume that a
> > database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
> > therefore we have to shut down the server to do it), but I don't get why
> > that's the case, particularly if we support any kind of mixed setup
> > where there's some data that's encrypted and some that isn't, and since
> > we're talking about using different keys for different things, it seems
> > to me that we almost certainly should be able to easily say "well,
> > there's no key for this, so just don't go through the decrypt/encrypt
> > routines".
>
> No, we can't easily do different keys for different things since all the
> keys have to be online for crash recovery, so there isn't much value to
> having different keys since they always have to be online.

Wasn't this already discussed?  We should have a master key and then
additional keys for different tables, et al, which are encrypted with
the master key.  Joe, I believe, covered all this quite well.

Either way though, I don't think it really goes against the point that I
was trying to make- we should be able to figure out if a table is
encrypted or not based on some information that we arrange to have
available during crash recovery and online processing, and the absence
of such should allow us to skip the encryption/decryption routines and
work with the table as we do today.  We should be thinking about
migrating from a completely unencrypted database to a database which has
all the 'core' bits encrypted (possibly as part of pg_upgrade or through
an offline tool of some kind) but the user data not encrypted, and then
online allow users to create new tables which are encrypted (maybe by
putting them in a particular tablespace or as a single table) and then
operate with those tables just like they would any other table in the
system, and let them manage how they move their sensitive data from some
other table into the encrypted table (or from another system into the
encrypted table).

Thanks,

Stephen

On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote:
> Masahiko Sawada copied this section as a desired direction, so I want to
> drill down into it.  I think we have five possible approaches for level
> 3 listed above.
> 
> The simplest approach would be to say that the LSN/page-number and WAL
> segment-number used as IV will not collide and we can just use them
> directly.

Looking at the bits we have, the IV for AES is 16 bytes.  Since we know
we have to use LSN (to change the IV for each page write), and the page
number (so WAL updates that change multiple pages with the same LSN use
different IVs), that uses 12 bytes:

    LSN         8 bytes
    page-number 4 bytes

That leaves 4 bytes unused.  If we use CTR, we need 11 bits for the
counter to support 32k pages sizes (per Sehrope Sarkuni), and we can use
the remaining 5 bits as constants to indicate heap, index, or WAL. 
(Technically, since we are not encrypting the first 16 bytes, we could
use one less bit for the counter.)  If we also use relfilenode, that is
4 bytes, so we have no bits for the heap/index/WAL constant, and no
space for the CTR counter, meaning we would have to use CBC mode.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote:
> > You are right that we can allow it online, but we haven't been
> > discussing these cases since it is easy to do this because we have
> > access to the keys.  I do think whatever code we use for checksum online
> > changes will be used for encryption online changes.  We would need a
> > per-page bit to indicate encryption, hopefully in the first 16 bytes.
> 
> Arranging to have an individual table move from being plain to
> encrypted is something that would be nice to support in an online and
> non-blocking manner, but *that* is a bunch of additional work that we
> don't need to do today as opposed to being something that's just part of
> the initial design.  Sure, it might use functions/capabilities that
> pg_checksums also use, but I don't know that we need to think about the
> code sharing there being much more than that, just that those
> capabilities should be built out in a way that they can be used for
> multiple things (and based on what I saw, that looks like it's exactly
> how that code was being written already).

Yes, we need to see how we are going to do that for checksums and
encryption and come up with a plan.

> > > There seems to be a strong thrust on this thread to assume that a
> > > database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
> > > therefore we have to shut down the server to do it), but I don't get why
> > > that's the case, particularly if we support any kind of mixed setup
> > > where there's some data that's encrypted and some that isn't, and since
> > > we're talking about using different keys for different things, it seems
> > > to me that we almost certainly should be able to easily say "well,
> > > there's no key for this, so just don't go through the decrypt/encrypt
> > > routines".
> > 
> > No, we can't easily do different keys for different things since all the
> > keys have to be online for crash recovery, so there isn't much value to
> > having different keys since they always have to be online.
> 
> Wasn't this already discussed?  We should have a master key and then
> additional keys for different tables, et al, which are encrypted with
> the master key.  Joe, I believe, covered all this quite well.

Yes, I am disagreeing with that.  I posted a 5-option email that went
over the issue and explored the value in different keys.  I am still
unclear of the benefit since it doesn't fix the 68GB limit unless we do
it per 1GB file, and we don't even know if that limit is per key or per
key/IV combo.  We can't move ahead until we decide that.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Greetings,

* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote:
> > > You are right that we can allow it online, but we haven't been
> > > discussing these cases since it is easy to do this because we have
> > > access to the keys.  I do think whatever code we use for checksum online
> > > changes will be used for encryption online changes.  We would need a
> > > per-page bit to indicate encryption, hopefully in the first 16 bytes.
> >
> > Arranging to have an individual table move from being plain to
> > encrypted is something that would be nice to support in an online and
> > non-blocking manner, but *that* is a bunch of additional work that we
> > don't need to do today as opposed to being something that's just part of
> > the initial design.  Sure, it might use functions/capabilities that
> > pg_checksums also use, but I don't know that we need to think about the
> > code sharing there being much more than that, just that those
> > capabilities should be built out in a way that they can be used for
> > multiple things (and based on what I saw, that looks like it's exactly
> > how that code was being written already).
>
> Yes, we need to see how we are going to do that for checksums and
> encryption and come up with a plan.

This is already being done though- Andres has a patch posted already and
my recollection from a quick look at that is that it should work just
fine for enabling checksums as well as potentially moving a table to be
encrypted online- the main common bit being that we need a way to say
"OK, everything has been done but we need to flip this flag and make
sure that everyone knows that this is now all checksum'd or all
encrypted".  The only thing there that I'm not sure about is that when
it comes to checksums, I believe the idea is that it's cluster-wide,
while with encryption that would only be true if we were trying to do
something like move the entire cluster from unencrypted to encrypted in
an online fashion (including WAL, CLOG, et al...) and if that's the case
then there's a bunch of other complicated bits, I believe, that we'd
have to work out, and I don't really think it's necessary or sensible to
worry about that right now.  Those are problems we don't currently have
with checksums either- the WAL already has them and I don't think
anyone's trying to address the fact that other rather core pieces of
the system don't currently.

> > > > There seems to be a strong thrust on this thread to assume that a
> > > > database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
> > > > therefore we have to shut down the server to do it), but I don't get why
> > > > that's the case, particularly if we support any kind of mixed setup
> > > > where there's some data that's encrypted and some that isn't, and since
> > > > we're talking about using different keys for different things, it seems
> > > > to me that we almost certainly should be able to easily say "well,
> > > > there's no key for this, so just don't go through the decrypt/encrypt
> > > > routines".
> > >
> > > No, we can't easily do different keys for different things since all the
> > > keys have to be online for crash recovery, so there isn't much value to
> > > having different keys since they always have to be online.
> >
> > Wasn't this already discussed?  We should have a master key and then
> > additional keys for different tables, et al, which are encrypted with
> > the master key.  Joe, I believe, covered all this quite well.
>
> Yes, I am disagreeing with that.  I posted a 5-option email that went
> over the issue and explored the value in different keys.  I am still
> unclear of the benefit since it doesn't fix the 68GB limit unless we do
> it per 1GB file, and we don't even know if that limit is per key or per
> key/IV combo.  We can't move ahead until we decide that.

I understand the 68G limit that you're referring to to be key/IV combo,
which means that a key per relation should be just fine.

Even if it was per key, and it means having a key per 1GB file,
that wouldn't change the point that I was making, so I'm not entirely
sure why it's being mentioned in this context.

I disagree with any approach that lacks a master key with additional
sub-keys, if that helps clarify things.

Thanks,

Stephen

On Thu, Jul 25, 2019 at 08:07:28PM -0400, Stephen Frost wrote:
> > Yes, we need to see how we are going to do that for checksums and
> > encryption and come up with a plan.
> 
> This is already being done though- Andres has a patch posted already and
> my recollection from a quick look at that is that it should work just
> fine for enabling checksums as well as potentially moving a table to be
> encrypted online- the main common bit being that we need a way to say
> "OK, everything has been done but we need to flip this flag and make
> sure that everyone knows that this is now all checksum'd or all
> encrypted".  The only thing there that I'm not sure about is that when
> it comes to checksums, I believe the idea is that it's cluster-wide,
> while with encryption that would only be true if we were trying to do
> something like move the entire cluster from unencrypted to encrypted in
> an online fashion (including WAL, CLOG, et al...) and if that's the case
> then there's a bunch of other complicated bits, I believe, that we'd
> have to work out, and I don't really think it's necessary or sensible to
> worry about that right now.  Those are problems we don't currently have
> with checksums either- the WAL already has them and I don't think
> anyone's trying to address the fact that other rather core pieces of
> the system don't currently.

OK,

> > > > > There seems to be a strong thrust on this thread to assume that a
> > > > > database MUST go from ALL DECRYPTED to ALL ENCRYPTED in one shot (and
> > > > > therefore we have to shut down the server to do it), but I don't get why
> > > > > that's the case, particularly if we support any kind of mixed setup
> > > > > where there's some data that's encrypted and some that isn't, and since
> > > > > we're talking about using different keys for different things, it seems
> > > > > to me that we almost certainly should be able to easily say "well,
> > > > > there's no key for this, so just don't go through the decrypt/encrypt
> > > > > routines".
> > > > 
> > > > No, we can't easily do different keys for different things since all the
> > > > keys have to be online for crash recovery, so there isn't much value to
> > > > having different keys since they always have to be online.
> > > 
> > > Wasn't this already discussed?  We should have a master key and then
> > > additional keys for different tables, et al, which are encrypted with
> > > the master key.  Joe, I believe, covered all this quite well.
> > 
> > Yes, I am disagreeing with that.  I posted a 5-option email that went
> > over the issue and explored the value in different keys.  I am still
> > unclear of the benefit since it doesn't fix the 68GB limit unless we do
> > it per 1GB file, and we don't even know if that limit is per key or per
> > key/IV combo.  We can't move ahead until we decide that.
> 
> I understand the 68G limit that you're referring to to be key/IV combo,
> which means that a key per relation should be just fine.

Yes, that is what I thought too.

> Even if it was per key, and it means having a key per 1GB file,
> that wouldn't change the point that I was making, so I'm not entirely
> sure why it's being mentioned in this context.

Because I thought we would use a single key for the entire cluster
(heap/index/WAL), and only use another key for encryption key rotation.

> I disagree with any approach that lacks a master key with additional
> sub-keys, if that helps clarify things.

We all know we need a passphrase that unlocks an encryption key.  Are
you saying we need per-object/table keys?  Why, other than for key
rotation?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote:
> On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian <bruce@momjian.us> wrote:
> 
>     Looking at the bits we have, the IV for AES is 16 bytes.  Since we know
>     we have to use LSN (to change the IV for each page write), and the page
>     number (so WAL updates that change multiple pages with the same LSN use
>     different IVs), that uses 12 bytes:
> 
>             LSN         8 bytes
>             page-number 4 bytes
> 
>     That leaves 4 bytes unused.  If we use CTR, we need 11 bits for the
>     counter to support 32k pages sizes (per Sehrope Sarkuni), and we can use
>     the remaining 5 bits as constants to indicate heap, index, or WAL.
>     (Technically, since we are not encrypting the first 16 bytes, we could
>     use one less bit for the counter.)  If we also use relfilenode, that is
>     4 bytes, so we have no bits for the heap/index/WAL constant, and no
>     space for the CTR counter, meaning we would have to use CBC mode.
> 
> 
> You can still use CTR mode and include those to make the key + IV unique by
> adding them to the derived key rather than the IV.
>
> The IV per-page would still be LSN + page-number (with the block number added
> as it's evaluated across the page) and the relfilenode, heap/index, database,
> and anything else to make it unique can be included in the HKDF to create the
> per-file derived key.

I thought if we didn't have to hash the stuff together we would be less
likely to get collisions with the IV.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +