Re: Internal key management system

On Wed, Oct 28, 2020 at 09:24:35PM +0900, Masahiko Sawada wrote:
> On Tue, 27 Oct 2020 at 20:34, Bruce Momjian <bruce@momjian.us> wrote:
> > You need to use separate keys for heap/index and WAL so you can
> > replicate to another server that uses a different heap/index key, but
> > the same WAL.  You can then fail-over to the replica and change the WAL
> > key to complete full key rotation.  The replication protocol needs to
> > decrypt, and the receiving end has to encrypt with a different
> > heap/index key.  This is the key rotation method this is planned.  This
> > is another good reason the keys should be in a separate directory so
> > they can be easily copied or replaced.
> 
> I think it's better we decrypt WAL in the xlogreader layer, instead of
> doing in replication protocol. That way, we also can support frontend
> tools that need to read WAL such as pg_waldump and pg_rewind as well
> as logical decoding.

Sure.  I was just saying the heap/index files coming from the primary
should have decrypted heap/index blocks, but I was not sure what level
it should happen.  If the data coming out the primary is encrypted, you
would need the old (to decrypt) and new (to encrypt) keys on the
standby, which seems too complex.

To clarify, the data and heap/index pages in the WAL are only encrypted
with the WAL key, but when pg_basebackup is streaming the files from
PGDATA, it shouldn't be encrypted, or encrypted only with the WAL key,
at the time of transfer since the receiver should be re-encrypting it. 
If that will not work, we should know now.

> > > I want to take a closer look at how the current implementation will
> > > play with physical replication. I assume the WAL and heap keys have to
> > > be constant for the full cluster lifetime, short of a dump and reload.
> >
> > The WAL key can change if you are willing to stop/start the server.  You
> > only read the WAL during crash recovery.
> 
> We might need to consider having multiple key generations, rather than
> in-place rotation. If we simply update the WAL key in-place in the
> primary, archive WALs restored via restore_command cannot be decrypted
> in the replica. We might need to do generation management for WAL key
> and provide the functionality to purge old WAL keys.

Since we have the keys stored in the file system, I think we will use a
command-line tool that can access both old and new keys and re-encrypted
the archived WAL.  I think old/new keys inside the server is too
complex.

> > The idea of the SQL key was to give the boot key a use, but I am now
> > seeing that the SQL key is just holding us back, and is preventing the
> > boot testing that is a requirement.  Maybe we just need to forget the
> > SQL part and focus on the TDE usage now, and come back to the SQL part.
> > I am also not 100% clear on the usefulness of the SQL key.
> 
> I agree to focus on the TDE usage now.

I admit the SQL key idea was mine, and I now see it was a bad idea since
it just adds confusion and doesn't add value.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee