AW: Encryption in pg_dump

AW: Encryption in pg_dump

AW: Encryption in pg_dump

Hi,

i had to do this on the fly compression 30 years ago in Oracle.
You had to use a pipe construct to do this.
Oracle was not able to do this as it is concept in Unix/Linux.

I am looking for an on the fly compression for a year now for PG 
I didnt't dig very deep as i spent most of the time with MySQL/MariaDB.

So i guess this is really a FAQ.

If you have a solution please share,

best,

Anton


-----Ursprüngliche Nachricht-----
Von: Bruce Momjian  
Gesendet: Mittwoch, 22. Juli 2020 20:56
An: Paul Förster 
Cc: vinod kale ; pgsql-admin@postgresql.org
Betreff: Re: Encryption in pg_dump

On Mon, Jul 20, 2020 at 03:24:16PM +0200, Paul Förster wrote:
> Hi Vinod,
> 
> > On 20. Jul, 2020, at 14:40, vinod kale  wrote:
> > 
> > Hello Team,
> > 
> > I am using postgresql 10.8 version in our prod environment. We take dump backup on daily basis of full database. And this dump backup creates in human readable format. 
> > My requirement is backup should creates in encrypted format at the time of pg_dump running. 
> > Is there any way or any option available in pg_dump that will creates the backup in encrypted format or any tool available for same.
> 
> sorry, no way. The only way you have is to pipe the output of pg_dump through some encryption tool. pg_dump does not offer to create encrypted dumps directly using some command line option.

Does anyone know why we are getting so many requests for encrypting dumps all of a sudden?

	July 20  CAD7GeHRZ3tu8y9VG_0juVK3RQq6r+t6HciguEz_C8Eh85Hmq3A@mail.gmail.com
	July 10  CALLD-SGTbu0HCW8NGVg+ycC8PKn9opcbN7aw4CvLj3OVC-hw=Q@mail.gmail.com
	July  8  CAFiui6G0sgLoURsK61vmiPR1Amd7PQt82cvgdmZ7+ukdsbV1QQ@mail.gmail.com

-- 
  Bruce Momjian          https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee

Re: Encryption in pg_dump

Hi Tal,

> On 23. Jul, 2020, at 10:27, Tal Glo  wrote:
> There is a way to implement full homomorphic encryption (FHE) with Postgres.

ok.

> I've used a relatively old version (2.3.1) of Microsoft's SEAL library in my University project for that.

I don't know about Windows but I assume, Linux would be similar?

> 2. Handling queries related to FHE encrypted attributes on the server side requires an implementation of own C language functions.

we don't do that. Also, we have third party applications. In case they need to be modified, this is impossible.

> It's not always a good Idea to say that something cannot be done or that some one needs to be replaced. Sometimes it's worth to develop some new process, based on a mixture of available technologies out there.

right, but only if it makes sense. To put a wallet in the file system and its key right next to it does not make sense but only serves to make IT heads and security "gurus" go quiet. In these cases, only replacing them by competent ones helps.

Cheers,
Paul

Re: Encryption in pg_dump

On 7/22/20 1:55 PM, Bruce Momjian wrote:
> On Mon, Jul 20, 2020 at 03:24:16PM +0200, Paul Förster wrote:
>> Hi Vinod,
>>
>>> On 20. Jul, 2020, at 14:40, vinod kale  wrote:
>>>
>>> Hello Team,
>>>
>>> I am using postgresql 10.8 version in our prod environment. We take dump backup on daily basis of full database. And this dump backup creates in human readable format.
>>> My requirement is backup should creates in encrypted format at the time of pg_dump running.
>>> Is there any way or any option available in pg_dump that will creates the backup in encrypted format or any tool available for same.
>> sorry, no way. The only way you have is to pipe the output of pg_dump through some encryption tool. pg_dump does not offer to create encrypted dumps directly using some command line option.
> Does anyone know why we are getting so many requests for encrypting
> dumps all of a sudden?
>
> 	July 20  CAD7GeHRZ3tu8y9VG_0juVK3RQq6r+t6HciguEz_C8Eh85Hmq3A@mail.gmail.com
> 	July 10  CALLD-SGTbu0HCW8NGVg+ycC8PKn9opcbN7aw4CvLj3OVC-hw=Q@mail.gmail.com
> 	July  8  CAFiui6G0sgLoURsK61vmiPR1Amd7PQt82cvgdmZ7+ukdsbV1QQ@mail.gmail.com

Audit season?

-- 
Angular momentum makes the world go 'round.

AW: Encryption in pg_dump

Re: Encryption in pg_dump

On 7/23/20 4:05 AM, Paul Förster wrote:
> Hi Tal,
>
>> On 23. Jul, 2020, at 10:27, Tal Glo  wrote:
>> There is a way to implement full homomorphic encryption (FHE) with Postgres.
> ok.
>
>> I've used a relatively old version (2.3.1) of Microsoft's SEAL library in my University project for that.
> I don't know about Windows but I assume, Linux would be similar?
>
>> 2. Handling queries related to FHE encrypted attributes on the server side requires an implementation of own C language functions.
> we don't do that. Also, we have third party applications. In case they need to be modified, this is impossible.

That was my first thought, too.  We mostly run 3rd party applications, with 
very little home-grown.  TDE (transparent data encryption) is the only 
reasonable work-around.

>> It's not always a good Idea to say that something cannot be done or that some one needs to be replaced. Sometimes it's worth to develop some new process, based on a mixture of available technologies out there.
> right, but only if it makes sense. To put a wallet in the file system and its key right next to it does not make sense but only serves to make IT heads and security "gurus" go quiet. In these cases, only replacing them by competent ones helps.

Which is as unlikely as getting all 3rd party vendors to implement 
application-level encryption.

-- 
Angular momentum makes the world go 'round.

Re: Encryption in pg_dump

On Mon, Jul 20, 2020 at 03:24:16PM +0200, Paul Förster wrote:
> Hi Vinod,
> 
> > On 20. Jul, 2020, at 14:40, vinod kale  wrote:
> > 
> > Hello Team, 
> > 
> > I am using postgresql 10.8 version in our prod environment. We take dump backup on daily basis of full database. And this dump backup creates in human readable format. 
> > My requirement is backup should creates in encrypted format at the time of pg_dump running. 
> > Is there any way or any option available in pg_dump that will creates the backup in encrypted format or any tool available for same.
> 
> sorry, no way. The only way you have is to pipe the output of pg_dump through some encryption tool. pg_dump does not offer to create encrypted dumps directly using some command line option.

Does anyone know why we are getting so many requests for encrypting
dumps all of a sudden?

	July 20  CAD7GeHRZ3tu8y9VG_0juVK3RQq6r+t6HciguEz_C8Eh85Hmq3A@mail.gmail.com
	July 10  CALLD-SGTbu0HCW8NGVg+ycC8PKn9opcbN7aw4CvLj3OVC-hw=Q@mail.gmail.com
	July  8  CAFiui6G0sgLoURsK61vmiPR1Amd7PQt82cvgdmZ7+ukdsbV1QQ@mail.gmail.com

-- 
  Bruce Momjian          https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee

Re: Encryption in pg_dump

On Wed, Jul 22, 2020 at 03:05:35PM -0500, Ron wrote:
> On 7/22/20 1:55 PM, Bruce Momjian wrote:
> > On Mon, Jul 20, 2020 at 03:24:16PM +0200, Paul Förster wrote:
> > > Hi Vinod,
> > > 
> > > > On 20. Jul, 2020, at 14:40, vinod kale  wrote:
> > > > 
> > > > Hello Team,
> > > > 
> > > > I am using postgresql 10.8 version in our prod environment. We take dump backup on daily basis of full database. And this dump backup creates in human readable format.
> > > > My requirement is backup should creates in encrypted format at the time of pg_dump running.
> > > > Is there any way or any option available in pg_dump that will creates the backup in encrypted format or any tool available for same.
> > > sorry, no way. The only way you have is to pipe the output of pg_dump through some encryption tool. pg_dump does not offer to create encrypted dumps directly using some command line option.
> > Does anyone know why we are getting so many requests for encrypting
> > dumps all of a sudden?
> > 
> > 	July 20  CAD7GeHRZ3tu8y9VG_0juVK3RQq6r+t6HciguEz_C8Eh85Hmq3A@mail.gmail.com
> > 	July 10  CALLD-SGTbu0HCW8NGVg+ycC8PKn9opcbN7aw4CvLj3OVC-hw=Q@mail.gmail.com
> > 	July  8  CAFiui6G0sgLoURsK61vmiPR1Amd7PQt82cvgdmZ7+ukdsbV1QQ@mail.gmail.com
> 
> Audit season?

LOL, maybe.  ;-)

-- 
  Bruce Momjian          https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee

Re: Encryption in pg_dump

Greetings,

* Tal Glo (glozmantal@gmail.com) wrote:
> There is a way to implement full homomorphic encryption (FHE) with
> Postgres. The decision which attributes to encrypt must be done at database
> planning step. It worth mentioning that FHE is good to implement, if one
> wants to be able to do server side supported mathematical operations on the
> encrypted data. If the data just needs to be stored in an encrypted form
> then using FHE might be a storage overkill. A dump of the database with FHE
> encrypted attributes preserves the encryption of the encrypted attributes.

Wasn't it relatively recently pointed out that having the ability to do
mathematical operations on the encrypted data ends up leading someone to
be able to deduce a fair bit of information much more easily than
actually breaking the encryption would involve..?

https://eprint.iacr.org/2016/775.pdf

Not finding it right now but I recall there being another paper which
basically pointed out that if you have context information about the
encrypted data (if it's credit card numbers, or SSNs, or such), along
with the ability to perform some operations (addition, ordering, etc)
that you could derive the real values without that much difficulty.

> 1. The encrypted attributes were big in size and Postgres cannot store the
> FHE cypher texts data type. 20,858 encrypted values used 3,6GB. ~33% of
> that size is because of base64 and the rest of the size ist because of my
> chosen encryption parameters.

A bit confused as to why you'd use base64 to store the data- would make
a lot more sense to use bytea.

> 2. Handling queries related to FHE encrypted attributes on the server side
> requires an implementation of own C language functions.

This could certainly be done in an extension, of course.

Thanks,

Stephen

Re: Encryption in pg_dump

Greetings,

* Ron (ronljohnsonjr@gmail.com) wrote:
> On 7/23/20 4:05 AM, Paul Förster wrote:
> >>On 23. Jul, 2020, at 10:27, Tal Glo  wrote:
> >>There is a way to implement full homomorphic encryption (FHE) with Postgres.
> >ok.
> >
> >>I've used a relatively old version (2.3.1) of Microsoft's SEAL library in my University project for that.
> >I don't know about Windows but I assume, Linux would be similar?
> >
> >>2. Handling queries related to FHE encrypted attributes on the server side requires an implementation of own C language functions.
> >we don't do that. Also, we have third party applications. In case they need to be modified, this is impossible.
> 
> That was my first thought, too.  We mostly run 3rd party applications, with
> very little home-grown.  TDE (transparent data encryption) is the only
> reasonable work-around.

TDE and pg_dump are pretty different things.  Encryption of backups
certainly makes sense (though pg_dump is really an export and not a
backup...) when they're being shipped off to somewhere else.  Of course,
you can do that with gpg or similar for pg_dump, though I wouldn't be
against adding support directly.  Ideally, we'd do that in a way that
leverages the code we already have for encryption, but to do that would
require moving more code into the common lib.

As for TDE, I agree that it's that same problem that using it means
that, while the data is encrypted on the disk, the key is available in
memory, and therefore the attack vectors that it actually helps with are
pretty few...  If the concern is an online compromise of the system,
then the attacker will have the encrypted data and the key, so TDE isn't
helpful in that case.  If the attack vector is an offline data-at-rest
attack, then encryption lower down the stack (either filesystem
encryption with fscrypto, or block-level encryption) can be used to
address it.  Filesystem or block-level encryption also addresses the
attack vector of "the attacker broke into our SAN", at least if the keys
aren't ever stored anywhere on the SAN.

> >>It's not always a good Idea to say that something cannot be done or that some one needs to be replaced. Sometimes it's worth to develop some new process, based on a mixture of available technologies out there.
> >right, but only if it makes sense. To put a wallet in the file system and its key right next to it does not make sense but only serves to make IT heads and security "gurus" go quiet. In these cases, only replacing them by competent ones helps.
> 
> Which is as unlikely as getting all 3rd party vendors to implement
> application-level encryption.

That is a bit unfortuante.  I do like the idea that some have had of
adding options into the library (eg: libpq) to support client side
encryption of the data before it's sent to the database server, to
address this.  That makes the encryption happen as early on as possible
(at least, for us) while ideally making it work with 3rd party
applications- the issue here is that for such things to work you can't
actually encrypt *everything* otherwise indexes and such aren't any
good, so you have to have enough knowledge to be able to define what
should and shouldn't be encrypted, etc...

Thanks,

Stephen

Re: Encryption in pg_dump

Greetings,

* Ron (ronljohnsonjr@gmail.com) wrote:
> TDE, though, protects against hackers who break in and try to slurp out as
> much data as possible.  Still pretty unlikely, but within the realm of
> reason.

That's an unfortunate misconception- TDE won't help with that as the key
will be in the server's memory and therefore accessible to the attacker.

Certainly, if the server is doing the decryption and is compromised in
an online fashion, then the attacker is going to have access to that
data.

Thanks,

Stephen

Re: Encryption in pg_dump

Hi Vinod,

> On 20. Jul, 2020, at 14:40, vinod kale  wrote:
> 
> Hello Team, 
> 
> I am using postgresql 10.8 version in our prod environment. We take dump backup on daily basis of full database. And this dump backup creates in human readable format. 
> My requirement is backup should creates in encrypted format at the time of pg_dump running. 
> Is there any way or any option available in pg_dump that will creates the backup in encrypted format or any tool available for same.

sorry, no way. The only way you have is to pipe the output of pg_dump through some encryption tool. pg_dump does not offer to create encrypted dumps directly using some command line option.

Cheers,
Paul

Re: Encryption in pg_dump

Hi Tal,

please don't top-post.

> On 23. Jul, 2020, at 12:20, Tal Glo  wrote:
> 
> You are correct. If third party applications rely on the FHE encrypted data, then they will need to change the way they quire it. But if it's companies sensitive internal data stored outside (e.g. in a cloud) and the company ist the only one that calculates on it, then it may be worth looking into that.

as I said, that's impossible. And we don't have anything in the cloud and never will.

> I didn't use Windows at all, everything ran on Linux.

ok.

Cheers,
Paul

Re: Encryption in pg_dump

> On Jul 23, 2020, at 9:34 AM, Rui DeSousa  wrote:
> 
> Encryption at rest also protects from warrantee service where you’re sending a failed drive back to a vendor. I could see the drive being refurbished and send out with readable data on it if not encrypted. 

And potentially allows you to reuse the drive. And *if* the server/drive is stolen, in the US at least, means you don't have a breach and don't have to notify customers.

Re: Encryption in pg_dump

On 7/23/20 8:52 AM, Scott Ribe wrote:
>> On Jul 22, 2020, at 11:29 PM, vinod kale  wrote:
>>
>> Actually our requirement is neither any of the data in readable format on server.
> Sounds like you need encryption at the disk or volume level.

Volume encryption only protects against stolen disks, which is... highly 
unlikely.  (The only scenarios I can think of are USB drives which get lost 
in transit -- think "station wagon full of tapes hurtling down the highway 
-- and law enforcement seizing the computers.)

TDE, though, protects against hackers who break in and try to slurp out as 
much data as possible.  Still pretty unlikely, but within the realm of reason.

-- 
Angular momentum makes the world go 'round.

Re: Encryption in pg_dump

Hi Ron,

> On 23. Jul, 2020, at 09:17, Ron  wrote:
> Adding hooks into libgpgme from pg_dump and pg_restore (needed for --format=directory) would be Very Helpful.

I guess, replacing stupid IT heads with competent ones makes more sense.

Cheers,
Paul

Re: Encryption in pg_dump

Hi Bruce,

> On 22. Jul, 2020, at 20:55, Bruce Momjian  wrote:
> 
> Does anyone know why we are getting so many requests for encrypting
> dumps all of a sudden?

probably because a) people don't read past posts and b) more and more IT heads decide that *everything*, be it internal to the company or not, has to be encrypted.

Cheers,
Paul

Re: Encryption in pg_dump

Paul Förster  writes:

> Hi Bruce,
>
>> On 22. Jul, 2020, at 20:55, Bruce Momjian  wrote:
>> 
>> Does anyone know why we are getting so many requests for encrypting
>> dumps all of a sudden?
>
> probably because a) people don't read past posts and b) more and more IT heads decide that *everything*, be it internal to the company or not, has to be encrypted.
>

Yes, I think the IT heads issue is the primary driver - combined with
very poor understanding of information security at senior levels and a
huge growth of poor quality and 'snake oil salesmen' in the IT security space
due to the amount of money ill informed senior managers are throwing at
what they think is a technical problem which usually is in fact a
business process problem. 

-- 
Tim Cross

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Hi Tim,

> On 23. Jul, 2020, at 07:34, Tim Cross  wrote:
> 
> Yes, I think the IT heads issue is the primary driver - combined with
> very poor understanding of information security at senior levels and a
> huge growth of poor quality and 'snake oil salesmen' in the IT security space
> due to the amount of money ill informed senior managers are throwing at
> what they think is a technical problem which usually is in fact a
> business process problem. 

and the best solution to that nonsense comes from the big red O. They use PKCS#12 wallets. Good, but as soon as you need auto-open wallets, which you do for every automated job like backups, you put the key right next to it. So it's like having a big safe in your house but the combination of the lock is written all over it. So much for security by obscurity.

For PostgreSQL we use a umask of 077 in our profile. That's why I keep telling my IT head that this is enough and it does not make sense to put the key next to the safe.

Cheers,
Paul

Re: Encryption in pg_dump

Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

Re: Encryption in pg_dump

> On Jul 23, 2020, at 8:56 AM, Stephen Frost  wrote:
> 
> Filesystem or block-level encryption also addresses the
> attack vector...

They also fix the "oops we were creating temp/log files locally which were not encrypted"

Re: Encryption in pg_dump

> On Jul 22, 2020, at 11:29 PM, vinod kale  wrote:
> 
> Actually our requirement is neither any of the data in readable format on server. 

Sounds like you need encryption at the disk or volume level.