Re: [COMMITTERS] pgsql: Add transforms feature

On 04/28/2015 04:10 PM, Andrew Dunstan wrote:
>
> On 04/28/2015 12:03 PM, Andrew Dunstan wrote:
>>
>> [switching to -hackers]
>>
>> On 04/28/2015 11:51 AM, Andrew Dunstan wrote:
>>>
>>> On 04/28/2015 09:04 AM, Michael Paquier wrote:
>>>> On Tue, Apr 28, 2015 at 10:01 AM, Michael Paquier
>>>> <michael.paquier@gmail.com> wrote:
>>>>> On Tue, Apr 28, 2015 at 9:55 AM, Andrew Dunstan wrote:
>>>>>> w.r.t MSVC builds, it looks like we need entries in 
>>>>>> $contrib_extraincludes
>>>>>> in src/tools/msvc/Mkvcbuild.pm at the very least.
>>>>> If our goal is to put back to green the Windows nodes as quick as
>>>>> possible, we could bypass their build this way , and we'll
>>>>> additionally need to update the install script and
>>>>> vcregress.pl/contribcheck to bypass those modules accordingly. Now I
>>>>> don't think that we should make the things produced inconsistent.
>>>> OK, attached are two patches to put back the buildfarm nodes using 
>>>> MSVC to green
>>>> - 0001 adds support for build and installation of the new transform
>>>> modules: hstore_plperl, hstore_plpython and  ltree_plpython. Note that
>>>> this patch is enough to put back the buildfarm to a green state for
>>>> MSVC, but it disables the regression tests for those modules,
>>>> something addressed in the next patch...
>>>> - 0002 adds support for regression tests for the new modules. The
>>>> thing is that we need to check the major version version of Python
>>>> available in configuration and choose what are the extensions to
>>>> preload before the tests run. It is a little bit hacky... But it has
>>>> the merit to work, and I am not sure we could have a cleaner solution
>>>> by looking at the Makefiles of the transform modules that use
>>>> currently conditions based on $(python_majorversion).
>>>> Regards,
>>>
>>>
>>> I have pushed the first of these with some tweaks.
>>>
>>> I'm looking at the second.
>>>
>>>
>>
>>
>> Regarding this second patch - apart from the buggy python version 
>> test which I just fixed in the first patch, I see this:
>>
>>    +       if ($pyver eq "2")
>>    +       {
>>    +           push @opts, "--load-extension=plpythonu";
>>    +           push @opts, '--load-extension=' . $module . 'u';
>>    +       }
>>
>>
>> But what do we do for Python3?
>>
>> Do we actually have a Windows machine building with Python3?
>
>
> The answer seems to be "probably not". When I tried enabling this with 
> bowerbird I got a ton of errors like these:
>
>    plpy_cursorobject.obj : error LNK2001: unresolved external symbol
>    PyObject_SelfIter [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
>    plpy_cursorobject.obj : error LNK2019: unresolved external symbol
>    __imp_PyType_Ready referenced in function PLy_cursor_init_type
>    [C:\prog\bf\root\HEAD\pgsql\plpython3.vcxproj]
>
>
> Something else to fix I guess.
>
>

OK, I fixed this as Michael suggested by installing the 64 bit python3 
(it's a pity that python.org didn't offer that to me as a download on 
their front page - that's a bit ugly).

However, when it came to running these tests that was a miserable 
failure. And indeed, we don't run any regression tests for plpython3 on 
MSVC.

So I committed this with just python2 tests enabled. All the buildfarm 
MSVC hosts seem to be using python2 anyway.

Meanwhile, we have some work to do on the mingw/gcc side.

These changes help make some progress - they let compilation succeed for 
hstore_plperl but I still get linkage failures. I suspect we need some 
things marked for export that we haven't to be marked needed before.

diff --git a/contrib/hstore_plperl/Makefile b/contrib/hstore_plperl/Makefile
index ddf6036..81fe1af 100644
--- a/contrib/hstore_plperl/Makefile
+++ b/contrib/hstore_plperl/Makefile
@@ -3,7 +3,7 @@ MODULE_big = hstore_plperl OBJS = hstore_plperl.o

-PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(perl_archlibexp)/CORE 
-I$(top_srcdir)/contrib/hstore
+PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plperl -I$(top_srcdir)/contrib/hstore
 EXTENSION = hstore_plperl hstore_plperlu DATA = hstore_plperl--1.0.sql hstore_plperlu--1.0.sql
@@ -21,3 +21,6 @@ top_builddir = ../.. include $(top_builddir)/src/Makefile.global include
$(top_srcdir)/contrib/contrib-global.mkendif
 
+
+override CPPFLAGS := $(CPPFLAGS) -DPLPERL_HAVE_UID_GID 
-I$(perl_archlibexp)/CORE
+override CFLAGS += -Wno-comment



The linkage failures look like:


x86_64-w64-mingw32-gcc -Wall -Wmissing-prototypes -Wpointer-arith 
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute 
-Wformat-security -fno-strict-aliasing -fwrapv 
-fexcess-precision=standard -g -O2  -Wno-comment -I../../src/pl/plperl 
-I../../contrib/hstore -I. -I. -I../../src/include 
-I./src/include/port/win32 -DEXEC_BACKEND 
-I/c/prog/3p64/include/libxml2  -I/c/prog/3p64/include 
-I/c/prog/3p64/openssl/include "-I../../src/include/port/win32" 
-DPLPERL_HAVE_UID_GID -IC:/Perl64/lib/CORE  -c -o hstore_plperl.o 
hstore_plperl.c
x86_64-w64-mingw32-gcc -Wall -Wmissing-prototypes -Wpointer-arith 
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute 
-Wformat-security -fno-strict-aliasing -fwrapv 
-fexcess-precision=standard -g -O2  -Wno-comment  -shared -static-libgcc 
-o hstore_plperl.dll  hstore_plperl.o -L../../src/port 
-L../../src/common -Wl,--allow-multiple-definition 
-Wl,--disable-auto-import -L/c/prog/3p64/lib  -L/c/prog/3p64/lib 
-L/c/prog/3p64/openssl/lib/VC -Wl,--as-needed   -L../../src/backend 
-lpostgres -lpgcommon -lpgport -lintl -lxslt -lxml2 -lz -lws2_32 -lm  
-lws2_32 -Wl,--export-all-symbols -Wl,--out-implib=libhstore_plperl.a
hstore_plperl.o: In function `hstore_to_plperl':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:17: 
undefined reference to `hstoreUpgrade'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:24: 
undefined reference to `__imp_Perl_get_context'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:24: 
undefined reference to `__imp_Perl_newSV_type'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:26: 
undefined reference to `__imp_Perl_hv_common_key_len'
hstore_plperl.o: In function `cstr2sv':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:115: 
undefined reference to `__imp_Perl_newSVpv'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:119: 
undefined reference to `__imp_Perl_newSVpv'
hstore_plperl.o: In function `hstore_to_plperl':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:32: 
undefined reference to `__imp_Perl_newSV'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:37: 
undefined reference to `__imp_Perl_newRV'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:57: 
undefined reference to `__imp_Perl_get_context'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:57: 
undefined reference to `__imp_Perl_hv_iterinit'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:59: 
undefined reference to `__imp_Perl_hv_iternext_flags'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:68: 
undefined reference to `hstoreCheckKeyLen'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:71: 
undefined reference to `__imp_Perl_newSVsv'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:89: 
undefined reference to `__imp_Perl_sv_2pvutf8'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:64:
undefined reference to `__imp_Perl_newSVpvn_flags'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:71: 
undefined reference to `__imp_Perl_newSVsv'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:80: 
undefined reference to `hstoreCheckValLen'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98: 
undefined reference to `__imp_Perl_sv_free'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98: 
undefined reference to `__imp_Perl_sv_free2'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:89: 
undefined reference to `__imp_Perl_sv_2pvutf8'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:87: 
undefined reference to `__imp_Perl_sv_2pv_flags'
hstore_plperl.o: In function `plperl_to_hstore':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:87: 
undefined reference to `hstoreUniquePairs'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/hstore_plperl.c:88: 
undefined reference to `hstorePairs'
hstore_plperl.o: In function `sv2cstr':
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98: 
undefined reference to `__imp_Perl_sv_free'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:98: 
undefined reference to `__imp_Perl_sv_free2'
c:\MinGW\msys\1.0\home\pgrunner\bf\root\HEAD\pgsql\contrib\hstore_plperl/../../src/pl/plperl/plperl_helpers.h:87: 
undefined reference to `__imp_Perl_sv_2pv_flags'
collect2.exe: error: ld returned 1 exit status


cheers

andrew