Обсуждение: pgsql: Use OpenSSL EVP API for symmetric encryption in pgcrypto.
Use OpenSSL EVP API for symmetric encryption in pgcrypto. The old "low-level" API is deprecated, and doesn't support hardware acceleration. And this makes the code simpler, too. Discussion: <561274F1.1030000@iki.fi> Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/5ff4a67f63fd6d3eb01ff9707d4674ed54a89f3b Modified Files -------------- contrib/pgcrypto/openssl.c | 554 +++++++++++++++------------------------------ 1 file changed, 178 insertions(+), 376 deletions(-)
Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: > Use OpenSSL EVP API for symmetric encryption in pgcrypto. BTW, "narwhal" seems to have a problem with this. Not very clear what, maybe an incompatibility with old openssl versions? regards, tom lane
On Tue, Oct 18, 2016 at 6:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: >> Use OpenSSL EVP API for symmetric encryption in pgcrypto. > > BTW, "narwhal" seems to have a problem with this. > Not very clear what, maybe an incompatibility with old openssl versions? Details are here: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=narwhal&dt=2016-10-17%2016%3A00%3A01 -- Michael
On 10/18/2016 12:28 AM, Tom Lane wrote: > Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: >> Use OpenSSL EVP API for symmetric encryption in pgcrypto. > > BTW, "narwhal" seems to have a problem with this. > Not very clear what, maybe an incompatibility with old openssl versions? Dave, what version of OpenSSL are 'narwhal' and 'baiji' using? I've tried to reproduce this on my laptop, by compiling different versions of OpenSSL, between 0.9.8beta1 and 0.9.8 head, but without success.. Googling for the failed test vector, I found this: https://www.mail-archive.com/openssl-users@openssl.org/msg68108.html It looks like the same issue, given the symptoms, but AFAICS the pgcrypto code is correctly calling EVP_EncryptInit_ex() twice. - Heikki
On Tue, Oct 18, 2016 at 9:10 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > On 10/18/2016 12:28 AM, Tom Lane wrote: >> >> Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: >>> >>> Use OpenSSL EVP API for symmetric encryption in pgcrypto. >> >> >> BTW, "narwhal" seems to have a problem with this. >> Not very clear what, maybe an incompatibility with old openssl versions? > > > Dave, what version of OpenSSL are 'narwhal' and 'baiji' using? I've tried to > reproduce this on my laptop, by compiling different versions of OpenSSL, > between 0.9.8beta1 and 0.9.8 head, but without success.. Insanely old ones, as reflects the age of those machines which haven't been touched in 10 years or so... Baiji: 0.9.8e Narwhal: 0.9.6b and for completeness: Mastodon: 0.9.8e -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> wrote: >On Tue, Oct 18, 2016 at 9:10 AM, Heikki Linnakangas <hlinnaka@iki.fi> >wrote: >> On 10/18/2016 12:28 AM, Tom Lane wrote: >>> >>> Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: >>>> >>>> Use OpenSSL EVP API for symmetric encryption in pgcrypto. >>> >>> >>> BTW, "narwhal" seems to have a problem with this. >>> Not very clear what, maybe an incompatibility with old openssl >versions? >> >> >> Dave, what version of OpenSSL are 'narwhal' and 'baiji' using? I've >tried to >> reproduce this on my laptop, by compiling different versions of >OpenSSL, >> between 0.9.8beta1 and 0.9.8 head, but without success.. > >Insanely old ones, as reflects the age of those machines which haven't >been touched in 10 years or so... > >Baiji: 0.9.8e >Narwhal: 0.9.6b Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal shouldn't even compile with 0.9.6. - Heikki
On Tue, Oct 18, 2016 at 12:18 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > > > On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> wrote: >>On Tue, Oct 18, 2016 at 9:10 AM, Heikki Linnakangas <hlinnaka@iki.fi> >>wrote: >>> On 10/18/2016 12:28 AM, Tom Lane wrote: >>>> >>>> Heikki Linnakangas <heikki.linnakangas@iki.fi> writes: >>>>> >>>>> Use OpenSSL EVP API for symmetric encryption in pgcrypto. >>>> >>>> >>>> BTW, "narwhal" seems to have a problem with this. >>>> Not very clear what, maybe an incompatibility with old openssl >>versions? >>> >>> >>> Dave, what version of OpenSSL are 'narwhal' and 'baiji' using? I've >>tried to >>> reproduce this on my laptop, by compiling different versions of >>OpenSSL, >>> between 0.9.8beta1 and 0.9.8 head, but without success.. >> >>Insanely old ones, as reflects the age of those machines which haven't >>been touched in 10 years or so... >> >>Baiji: 0.9.8e >>Narwhal: 0.9.6b > > Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal shouldn't even compile with 0.9.6. Oops, sorry - I found another copy in the Mingw installation that it must be using (Mingw/Msys was a mess back then). I only have the header files to go by, but it looks like the very first 0.9.8 release. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
Dave Page <dpage@postgresql.org> writes: > On Tue, Oct 18, 2016 at 12:18 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >> On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> wrote: >>> Baiji: 0.9.8e >>> Narwhal: 0.9.6b >> Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal shouldn't even compile with 0.9.6. > Oops, sorry - I found another copy in the Mingw installation that it > must be using (Mingw/Msys was a mess back then). I only have the > header files to go by, but it looks like the very first 0.9.8 release. Is it possible that there's a header-vs-executable version mismatch contributing to the problem here? (Although you'd think we'd have hit it before now, if so.) regards, tom lane
On Tue, Oct 18, 2016 at 1:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Dave Page <dpage@postgresql.org> writes: >> On Tue, Oct 18, 2016 at 12:18 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >>> On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> wrote: >>>> Baiji: 0.9.8e >>>> Narwhal: 0.9.6b > >>> Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal shouldn't even compile with 0.9.6. > >> Oops, sorry - I found another copy in the Mingw installation that it >> must be using (Mingw/Msys was a mess back then). I only have the >> header files to go by, but it looks like the very first 0.9.8 release. > > Is it possible that there's a header-vs-executable version mismatch > contributing to the problem here? (Although you'd think we'd have > hit it before now, if so.) We're not actually calling the openssl binary are we? There's an openssl binary only in the Msys installation (the really old version), and headers/libraries but no executable in the Mingw installation. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
On 10/18/2016 04:32 PM, Dave Page wrote: > On Tue, Oct 18, 2016 at 1:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Dave Page <dpage@postgresql.org> writes: >>> On Tue, Oct 18, 2016 at 12:18 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >>>> On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> wrote: >>>>> Baiji: 0.9.8e >>>>> Narwhal: 0.9.6b >> >>>> Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal shouldn't even compile with 0.9.6. >> >>> Oops, sorry - I found another copy in the Mingw installation that it >>> must be using (Mingw/Msys was a mess back then). I only have the >>> header files to go by, but it looks like the very first 0.9.8 release. >> >> Is it possible that there's a header-vs-executable version mismatch >> contributing to the problem here? (Although you'd think we'd have >> hit it before now, if so.) > > We're not actually calling the openssl binary are we? There's an > openssl binary only in the Msys installation (the really old version), > and headers/libraries but no executable in the Mingw installation. Nope. Any chance of updating the OpenSSL on that box, to see if it helps? I'd rather not spend time investigate this further, if it turns out to be a bug in an ancient version of OpenSSL. - Heikki
Dave Page <dpage@postgresql.org> writes: > On Tue, Oct 18, 2016 at 1:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Is it possible that there's a header-vs-executable version mismatch >> contributing to the problem here? (Although you'd think we'd have >> hit it before now, if so.) > We're not actually calling the openssl binary are we? There's an > openssl binary only in the Msys installation (the really old version), > and headers/libraries but no executable in the Mingw installation. Sorry, what I meant was header vs library mismatch. regards, tom lane
On Tue, Oct 18, 2016 at 2:34 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > On 10/18/2016 04:32 PM, Dave Page wrote: >> >> On Tue, Oct 18, 2016 at 1:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> >>> Dave Page <dpage@postgresql.org> writes: >>>> >>>> On Tue, Oct 18, 2016 at 12:18 PM, Heikki Linnakangas <hlinnaka@iki.fi> >>>> wrote: >>>>> >>>>> On 18 October 2016 12:52:14 EEST, Dave Page <dpage@postgresql.org> >>>>> wrote: >>>>>> >>>>>> Baiji: 0.9.8e >>>>>> Narwhal: 0.9.6b >>> >>> >>>>> Hang on, I removed support for OpenSSL < 0.9.8 a while ago. Narwhal >>>>> shouldn't even compile with 0.9.6. >>> >>> >>>> Oops, sorry - I found another copy in the Mingw installation that it >>>> must be using (Mingw/Msys was a mess back then). I only have the >>>> header files to go by, but it looks like the very first 0.9.8 release. >>> >>> >>> Is it possible that there's a header-vs-executable version mismatch >>> contributing to the problem here? (Although you'd think we'd have >>> hit it before now, if so.) >> >> >> We're not actually calling the openssl binary are we? There's an >> openssl binary only in the Msys installation (the really old version), >> and headers/libraries but no executable in the Mingw installation. > > > Nope. > > Any chance of updating the OpenSSL on that box, to see if it helps? I'd > rather not spend time investigate this further, if it turns out to be a bug > in an ancient version of OpenSSL. The latest OpenSSL version won't even configure on there. I wonder if it's finally time to retire that old machine... -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
Dave Page <dpage@postgresql.org> writes: > On Tue, Oct 18, 2016 at 9:10 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >> Dave, what version of OpenSSL are 'narwhal' and 'baiji' using? I've tried to >> reproduce this on my laptop, by compiling different versions of OpenSSL, >> between 0.9.8beta1 and 0.9.8 head, but without success.. > Insanely old ones, as reflects the age of those machines which haven't > been touched in 10 years or so... > Baiji: 0.9.8e > Narwhal: 0.9.6b > and for completeness: > Mastodon: 0.9.8e Interestingly, it's now apparent that all three of those critters are failing identically. I begin to suspect that it's somehow Windows-specific; if it were a bug in a particular OpenSSL version, seems like Heikki's testing described above would have reproduced it. The only other Windows machines in the buildfarm that are building with openssl are Andrew's bowerbird and frogmouth; they're both passing. I wonder what openssl version is on those. regards, tom lane
I wrote: > The only other Windows machines in the buildfarm that are building > with openssl are Andrew's bowerbird and frogmouth; they're both passing. > I wonder what openssl version is on those. What's the plan for getting the buildfarm back to green in the wake of this patch? If there isn't one, I'd like to request that the patch be reverted until such time as some effort can be put into it. Right now, these failures are preventing us from telling whether the recent postgres_fdw changes will pass everywhere. regards, tom lane
On 10/22/2016 12:11 PM, Tom Lane wrote: > I wrote: >> The only other Windows machines in the buildfarm that are building >> with openssl are Andrew's bowerbird and frogmouth; they're both passing. >> I wonder what openssl version is on those. > What's the plan for getting the buildfarm back to green in the wake > of this patch? If there isn't one, I'd like to request that the patch > be reverted until such time as some effort can be put into it. Right > now, these failures are preventing us from telling whether the recent > postgres_fdw changes will pass everywhere. > > bowerbird: 1.0.1d frogmouth: 1.0.0c They probably both should be uodated, but are both more modern than the failing machines by the look of it. cheers andrew
On 10/22/2016 07:11 PM, Tom Lane wrote: > I wrote: >> The only other Windows machines in the buildfarm that are building >> with openssl are Andrew's bowerbird and frogmouth; they're both passing. >> I wonder what openssl version is on those. > > What's the plan for getting the buildfarm back to green in the wake > of this patch? If there isn't one, I'd like to request that the patch > be reverted until such time as some effort can be put into it. Right > now, these failures are preventing us from telling whether the recent > postgres_fdw changes will pass everywhere. Dave, what are we going to do about this? I cannot reproduce a similar breakage on my system with OpenSSL 0.9.8 versions, and I don't have access to the boxes themselves, so I'm pretty much stuck in trying to debug the issue. And I wouldn't like to spend much time on that anyway, given that the OpenSSL versions they use are ancient, and all other buildfarm animals are happy. Any chance you could update the OpenSSL versions on those systems, or failing that, remove --with-openssl? - Heikki
Hi On Sun, Oct 23, 2016 at 8:22 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > On 10/22/2016 07:11 PM, Tom Lane wrote: >> >> I wrote: >>> >>> The only other Windows machines in the buildfarm that are building >>> with openssl are Andrew's bowerbird and frogmouth; they're both passing. >>> I wonder what openssl version is on those. >> >> >> What's the plan for getting the buildfarm back to green in the wake >> of this patch? If there isn't one, I'd like to request that the patch >> be reverted until such time as some effort can be put into it. Right >> now, these failures are preventing us from telling whether the recent >> postgres_fdw changes will pass everywhere. > > > Dave, what are we going to do about this? I cannot reproduce a similar > breakage on my system with OpenSSL 0.9.8 versions, and I don't have access > to the boxes themselves, so I'm pretty much stuck in trying to debug the > issue. And I wouldn't like to spend much time on that anyway, given that the > OpenSSL versions they use are ancient, and all other buildfarm animals are > happy. As I mentioned earlier in the thread, I couldn't get a modern version to configure at all on there. > Any chance you could update the OpenSSL versions on those systems, or > failing that, remove --with-openssl? Well I could. But really, how useful is that given that it's got to be pretty rare that people won't use OpenSSL in a build? Those machines are so old now that I don't think there's any merit in keeping them is there? -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
On 24 October 2016 12:33:49 EEST, Dave Page <dpage@postgresql.org> wrote: >Hi > >On Sun, Oct 23, 2016 at 8:22 PM, Heikki Linnakangas <hlinnaka@iki.fi> >wrote: >> Any chance you could update the OpenSSL versions on those systems, or >> failing that, remove --with-openssl? > >Well I could. But really, how useful is that given that it's got to be >pretty rare that people won't use OpenSSL in a build? Those machines >are so old now that I don't think there's any merit in keeping them is >there? I'm ok with removing them from the buildfarm, if you feel that way. - Heikki
On Mon, Oct 24, 2016 at 12:30 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > > > On 24 October 2016 12:33:49 EEST, Dave Page <dpage@postgresql.org> wrote: >>Hi >> >>On Sun, Oct 23, 2016 at 8:22 PM, Heikki Linnakangas <hlinnaka@iki.fi> >>wrote: >>> Any chance you could update the OpenSSL versions on those systems, or >>> failing that, remove --with-openssl? >> >>Well I could. But really, how useful is that given that it's got to be >>pretty rare that people won't use OpenSSL in a build? Those machines >>are so old now that I don't think there's any merit in keeping them is >>there? > > I'm ok with removing them from the buildfarm, if you feel that way. Andrew, Tom - any thoughts on removing them (just from head and future branches)? -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
Dave Page <dpage@postgresql.org> writes: > On Mon, Oct 24, 2016 at 12:30 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >> I'm ok with removing them from the buildfarm, if you feel that way. > Andrew, Tom - any thoughts on removing them (just from head and future > branches)? I'd prefer the solution of disabling --with-openssl. If you just nuke those members, we never will find out whether the recent postgres_fdw changes pass on them. Besides, the point of the buildfarm has never been for all members to have the same configuration options. regards, tom lane
On 10/24/2016 08:56 AM, Tom Lane wrote: > Dave Page <dpage@postgresql.org> writes: >> On Mon, Oct 24, 2016 at 12:30 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote: >>> I'm ok with removing them from the buildfarm, if you feel that way. >> Andrew, Tom - any thoughts on removing them (just from head and future >> branches)? > I'd prefer the solution of disabling --with-openssl. If you just nuke > those members, we never will find out whether the recent postgres_fdw > changes pass on them. Besides, the point of the buildfarm has never > been for all members to have the same configuration options. > > Seems reasonable. cheers andrew
On Mon, Oct 24, 2016 at 4:21 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > > On 10/24/2016 08:56 AM, Tom Lane wrote: >> >> Dave Page <dpage@postgresql.org> writes: >>> >>> On Mon, Oct 24, 2016 at 12:30 PM, Heikki Linnakangas <hlinnaka@iki.fi> >>> wrote: >>>> >>>> I'm ok with removing them from the buildfarm, if you feel that way. >>> >>> Andrew, Tom - any thoughts on removing them (just from head and future >>> branches)? >> >> I'd prefer the solution of disabling --with-openssl. If you just nuke >> those members, we never will find out whether the recent postgres_fdw >> changes pass on them. Besides, the point of the buildfarm has never >> been for all members to have the same configuration options. >> >> > > > > Seems reasonable. OK, I think I got the perl stuff right - running test builds now. -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
On 10/25/2016 10:26 AM, Dave Page wrote: > > OK, I think I got the perl stuff right - running test builds now. > Ping me if you need help. cheers andrew
On Tue, Oct 25, 2016 at 10:22 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > > On 10/25/2016 10:26 AM, Dave Page wrote: >> >> >> OK, I think I got the perl stuff right - running test builds now. >> > > > Ping me if you need help. Thanks - looks like I got it though, they're all green again :-) -- Dave Page PostgreSQL Core Team http://www.postgresql.org/
Dave Page <dpage@postgresql.org> writes: > On Mon, Oct 24, 2016 at 4:21 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> On 10/24/2016 08:56 AM, Tom Lane wrote: >>> I'd prefer the solution of disabling --with-openssl. If you just nuke >>> those members, we never will find out whether the recent postgres_fdw >>> changes pass on them. Besides, the point of the buildfarm has never >>> been for all members to have the same configuration options. >> Seems reasonable. > OK, I think I got the perl stuff right - running test builds now. Hey Dave, baiji has been failing the REL_10_STABLE branch since you started running it there. I'm guessing you set it up to suppress --with-openssl only for HEAD, and not for $branch ge 'REL_10' ... regards, tom lane -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
On Fri, Oct 6, 2017 at 5:07 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dave Page <dpage@postgresql.org> writes:
> On Mon, Oct 24, 2016 at 4:21 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
>> On 10/24/2016 08:56 AM, Tom Lane wrote:
>>> I'd prefer the solution of disabling --with-openssl. If you just nuke
>>> those members, we never will find out whether the recent postgres_fdw
>>> changes pass on them. Besides, the point of the buildfarm has never
>>> been for all members to have the same configuration options.
>> Seems reasonable.
> OK, I think I got the perl stuff right - running test builds now.
Hey Dave, baiji has been failing the REL_10_STABLE branch since you
started running it there. I'm guessing you set it up to suppress
--with-openssl only for HEAD, and not for $branch ge 'REL_10' ...
Argh, yes. Should be fixed now.
Thanks.