Обсуждение: PostgreSQL and big data - FDW
Hi
I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster which uses KERBEROS authentication.
is it possible to achieve this ? which FDW should be used ?
Thanks in advance
Best Regards
Didier ROS
EDF
Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.
If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.
E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:
> Hi
>
> I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster
> which uses KERBEROS authentication.
>
> is it possible to achieve this ? which FDW should be used ?
Well, I would use the Hadoop FDW:
https://github.com/EnterpriseDB/hdfs_fdw
and it only supports these authentication methods:
Authentication Support
The FDW supports NOSASL and LDAP authentication modes. In order to use
NOSASL do not specify any OPTIONS while creating user mapping. For LDAP
username and password must be specified in OPTIONS while creating user mapping.
Not every FDW supports every Postgres server authentication method.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Hi Bruce
In the following link :
https://www.enterprisedb.com/blog/connecting-hadoop-and-edb-postgres-shrink-big-data-challenges
We can see :
"Support for various authentication methods (i.e. Kerberos, NOSASL, etc.)"
So HDFS_FDW support kerberos authentication . how to be sure of that ?
Could EDB make a clear statement on this point?
If so, how to implement this method ? is there any document on this subject ?
Thanks in advance.
Best Regards
Didier ROS
didier.ros@edf.fr
Tél. : +33 6 49 51 11 88
-----Message d'origine-----
De : bruce@momjian.us [mailto:bruce@momjian.us]
Envoyé : mercredi 24 juin 2020 11:13
À : ROS Didier <didier.ros@edf.fr>
Cc : pgsql-hackers@lists.postgresql.org
Objet : Re: PostgreSQL and big data - FDW
On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote:
> Hi
>
> I would like to use a Foreign Data Wrapper (FDW) to connect to a
> HADOOP cluster which uses KERBEROS authentication.
>
> is it possible to achieve this ? which FDW should be used ?
Well, I would use the Hadoop FDW:
https://github.com/EnterpriseDB/hdfs_fdw
and it only supports these authentication methods:
Authentication Support
The FDW supports NOSASL and LDAP authentication modes. In order to use
NOSASL do not specify any OPTIONS while creating user mapping. For LDAP
username and password must be specified in OPTIONS while creating user mapping.
Not every FDW supports every Postgres server authentication method.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires
etles informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa
destination,toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.
Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le
divulguerou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre
système,ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous
remercionségalement d'en avertir immédiatement l'expéditeur par retour du message.
Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont
sécuriséesou dénuées de toute erreur ou virus.
____________________________________________________
This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in
thisMessage is confidential. Any use of information contained in this Message not in accord with its purpose, any
disseminationor disclosure, either whole or partial, is prohibited except formal approval.
If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this
messagein error, please delete it and all copies from your system and notify the sender immediately by return message.
E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
On Wed, Jun 24, 2020 at 6:09 PM ROS Didier <didier.ros@edf.fr> wrote:
Hi Bruce
In the following link : https://www.enterprisedb.com/blog/connecting-hadoop-and-edb-postgres-shrink-big-data-challenges
We can see :
"Support for various authentication methods (i.e. Kerberos, NOSASL, etc.)"
So HDFS_FDW support kerberos authentication . how to be sure of that ?
Could EDB make a clear statement on this point?
HDFS_FDW does not support kerberos authentication.
The sentence you have pasted above is from the wish list or say TODO
list, here is what it says:
"Currently the HDFS_FDW only provides READ capabilities but EDB is planning the following additional functionality:"
The functionality was not implemented. I think the part of confusion might be
due to the formatting of the list in the blog.
You can follow the README[1] of HDFS_FDW to get an idea of how to use it.
Regards,
Jeevan
Greetings, * Bruce Momjian (bruce@momjian.us) wrote: > On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote: > > I would like to use a Foreign Data Wrapper (FDW) to connect to a HADOOP cluster > > which uses KERBEROS authentication. Sadly, not really. > > is it possible to achieve this ? which FDW should be used ? > > Well, I would use the Hadoop FDW: > > https://github.com/EnterpriseDB/hdfs_fdw > > and it only supports these authentication methods: > > Authentication Support > > The FDW supports NOSASL and LDAP authentication modes. In order to use > NOSASL do not specify any OPTIONS while creating user mapping. For LDAP > username and password must be specified in OPTIONS while creating user mapping. > > Not every FDW supports every Postgres server authentication method. That isn't really the issue here, the problem is really that the GSSAPI support in PG today doesn't support credential delegation- if it did, then the HDFS FDW (and the postgres FDW) could be easily extended to leverage those delegated credentials to connect. That's been something that's been on my personal todo list of things to work on but unfortunately I've not, as yet, had time to go implement. I don't actually think it would be very hard- if someone writes it, I'd definitely review it. Thanks, Stephen
Вложения
Hi Stephen My EDF company is very interested in this feature (KERBEROS authentication method and hdfs_fdw ). Is it possible to know how many days of development does this represent ? who can develop this implementation ? what cost? Best Regards Didier ROS EDF -----Message d'origine----- De : sfrost@snowman.net [mailto:sfrost@snowman.net] Envoyé : mercredi 24 juin 2020 18:53 À : Bruce Momjian <bruce@momjian.us> Cc : ROS Didier <didier.ros@edf.fr>; pgsql-hackers@lists.postgresql.org Objet : Re: PostgreSQL and big data - FDW Greetings, * Bruce Momjian (bruce@momjian.us) wrote: > On Wed, Jun 24, 2020 at 09:05:30AM +0000, ROS Didier wrote: > > I would like to use a Foreign Data Wrapper (FDW) to connect to a > > HADOOP cluster which uses KERBEROS authentication. Sadly, not really. > > is it possible to achieve this ? which FDW should be used ? > > Well, I would use the Hadoop FDW: > > https://github.com/EnterpriseDB/hdfs_fdw > > and it only supports these authentication methods: > > Authentication Support > > The FDW supports NOSASL and LDAP authentication modes. In order to use > NOSASL do not specify any OPTIONS while creating user mapping. For LDAP > username and password must be specified in OPTIONS while creating user mapping. > > Not every FDW supports every Postgres server authentication method. That isn't really the issue here, the problem is really that the GSSAPI support in PG today doesn't support credential delegation-if it did, then the HDFS FDW (and the postgres FDW) could be easily extended to leverage those delegated credentialsto connect. That's been something that's been on my personal todo list of things to work on but unfortunately I've not, as yet, had timeto go implement. I don't actually think it would be very hard- if someone writes it, I'd definitely review it. Thanks, Stephen Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires etles informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination,toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguerou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système,ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercionségalement d'en avertir immédiatement l'expéditeur par retour du message. Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécuriséesou dénuées de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in thisMessage is confidential. Any use of information contained in this Message not in accord with its purpose, any disseminationor disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this messagein error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
On Thu, Jun 25, 2020 at 07:02:37AM +0000, ROS Didier wrote: > Hi Stephen > > My EDF company is very interested in this feature (KERBEROS authentication method and hdfs_fdw ). > Is it possible to know how many days of development does this represent ? who can develop this implementation ? what cost? Uh, the only thing I can suggest is to contact one of the larger Postgres support companies (ones that have developers who understand the server code, or at least the FDW code), and ask them for estimates. The community really can't supply any of that, unless you want to do the work and want source code tips. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EnterpriseDB https://enterprisedb.com The usefulness of a cup is in its emptiness, Bruce Lee