Обсуждение: plperl on windows
Hi, For https://postgr.es/m/20211001222752.wrz7erzh4cajvgp6@alap3.anarazel.de I was trying to enable plperl on windows. And run into quite a few roadblocks - enough that I gave up. 1) plperl doesn't build against a modern-ish perl. The fix for that seems easy enough: https://postgr.es/m/20200501134711.08750c5f@antares.wagner.home 2) For some reason src/tools/install.pl doesn't install plperl[u].control, plperl[u]--1.0.sql - But apparently the buildfarm doesn't have that issue, because drongo successfully ran the plperl tests? 3) When building against strawberry perl 5.32.1.1 I see errors when loading plperl 4) When building against strawberry perl 5.30.3.1 I see a crash during execution of very simple statements [1] 5) Finally when building against strawberry perl 5.28.2.1, plperl kinda works. But there's a lot of regression test failures, many of them seemingly around error trapping. I saw that there's also active state perl, but it seems to require clicking through some terms and conditions for every download that I don't want to agree to. Greetings, Andres Freund [1] Exception thrown at 0x000000006FD75DB8 (perl530.dll) in postgres.exe: 0xC0000005: Access violation reading location 0x0000000000000008. perl530.dll!Perl_mg_get() + 56 bytes Unknown plperl.dll!select_perl_context(bool trusted) Line 667 C plperl.dll!plperl_inline_handler(FunctionCallInfoBaseData * fcinfo) Line 1941 C plperl.dll!plperlu_inline_handler(FunctionCallInfoBaseData * fcinfo) Line 2064 C postgres.exe!FunctionCall1Coll(FmgrInfo * flinfo, unsigned int collation, unsigned __int64 arg1) Line 1138 C postgres.exe!OidFunctionCall1Coll(unsigned int functionId, unsigned int collation, unsigned __int64 arg1) Line 1417 C postgres.exe!ExecuteDoStmt(ParseState * pstate, DoStmt * stmt, bool atomic) Line 2146 C postgres.exe!standard_ProcessUtility(PlannedStmt * pstmt, const char * queryString, bool readOnlyTree, ProcessUtilityContextcontext, ParamListInfoData * params, QueryEnvironment * queryEnv, _DestReceiver * dest, QueryCompletion* qc) Line 712 C postgres.exe!ProcessUtility(PlannedStmt * pstmt, const char * queryString, bool readOnlyTree, ProcessUtilityContextcontext, ParamListInfoData * params, QueryEnvironment * queryEnv, _DestReceiver * dest, QueryCompletion* qc) Line 530 C postgres.exe!PortalRunUtility(PortalData * portal, PlannedStmt * pstmt, bool isTopLevel, bool setHoldSnapshot, _DestReceiver* dest, QueryCompletion * qc) Line 1157 C postgres.exe!PortalRunMulti(PortalData * portal, bool isTopLevel, bool setHoldSnapshot, _DestReceiver * dest, _DestReceiver* altdest, QueryCompletion * qc) Line 1306 C postgres.exe!PortalRun(PortalData * portal, long count, bool isTopLevel, bool run_once, _DestReceiver * dest, _DestReceiver* altdest, QueryCompletion * qc) Line 790 C postgres.exe!exec_simple_query(const char * query_string) Line 1222 C postgres.exe!PostgresMain(const char * dbname, const char * username) Line 4499 C postgres.exe!BackendRun(Port * port) Line 4561 C postgres.exe!SubPostmasterMain(int argc, char * * argv) Line 5066 C postgres.exe!main(int argc, char * * argv) Line 190 C postgres.exe!invoke_main() Line 79 C++ postgres.exe!__scrt_common_main_seh() Line 288 C++ postgres.exe!__scrt_common_main() Line 331 C++ postgres.exe!mainCRTStartup(void * __formal) Line 17 C++ kernel32.dll!BaseThreadInitThunk() Unknown ntdll.dll!RtlUserThreadStart() Unknown [2] --- C:/Users/anfreund/src/postgres/src/pl/plperl/expected/plperl.out 2021-03-02 00:29:34.416742000 -0800 +++ C:/Users/anfreund/src/postgres/src/pl/plperl/results/plperl.out 2021-10-04 14:31:45.773612500 -0700 @@ -660,8 +660,11 @@ return $result; $$ LANGUAGE plperl; SELECT perl_spi_prepared_bad(4.35) as "double precision"; -ERROR: type "does_not_exist" does not exist at line 2. -CONTEXT: PL/Perl function "perl_spi_prepared_bad" + double precision +------------------ + +(1 row) + -- Test with a row type CREATE OR REPLACE FUNCTION perl_spi_prepared() RETURNS INTEGER AS $$ my $x = spi_prepare('select $1::footype AS a', 'footype'); @@ -696,37 +699,28 @@ NOTICE: This is a test -- check that restricted operations are rejected in a plperl DO block DO $$ system("/nonesuch"); $$ LANGUAGE plperl; -ERROR: 'system' trapped by operation mask at line 1. -CONTEXT: PL/Perl anonymous code block ... --- C:/Users/anfreund/src/postgres/src/pl/plperl/expected/plperl_plperlu.out 2021-03-02 00:29:34.425742300 -0800 +++ C:/Users/anfreund/src/postgres/src/pl/plperl/results/plperl_plperlu.out 2021-10-04 14:31:48.065612400 -0700 @@ -10,11 +10,17 @@ return 1; $$ LANGUAGE plperlu; -- compile plperlu code SELECT * FROM bar(); -- throws exception normally (running plperl) -ERROR: syntax error at or near "invalid" at line 4. -CONTEXT: PL/Perl function "bar" + bar +----- + +(1 row) + SELECT * FROM foo(); -- used to cause backend crash (after switching to plperlu) -ERROR: syntax error at or near "invalid" at line 4. at line 2. -CONTEXT: PL/Perl function "foo" + foo +----- + 1 +(1 row) + -ERROR: Unable to load Errno.pm into plperl at line 2. -BEGIN failed--compilation aborted at line 2. +ERROR: didn't get a CODE reference from compiling function "use_plperl" CONTEXT: compilation of PL/Perl function "use_plperl" -- make sure our overloaded require op gets restored/set correctly select use_plperlu(); @@ -86,6 +91,5 @@ AS $$ use Errno; $$; -ERROR: Unable to load Errno.pm into plperl at line 2. -BEGIN failed--compilation aborted at line 2. +ERROR: didn't get a CODE reference from compiling function "use_plperl" CONTEXT: compilation of PL/Perl function "use_plperl"
Hi, On 2021-10-04 14:38:16 -0700, Andres Freund wrote: > 2) For some reason src/tools/install.pl doesn't install plperl[u].control, > plperl[u]--1.0.sql - But apparently the buildfarm doesn't have that issue, > because drongo successfully ran the plperl tests? Oh, figured that one out: Install.pm checks the current directory for config.pl - but my invocation was from the source tree root (which is supported for most things). Because of that it skipped installing plperl, as it though it wasn't installed. Greetings, Andres Freund
Hi, On 2021-10-04 14:38:16 -0700, Andres Freund wrote: > 3) When building against strawberry perl 5.32.1.1 I see errors when loading > plperl > > 4) When building against strawberry perl 5.30.3.1 I see a crash during > execution of very simple statements [1] > > 5) Finally when building against strawberry perl 5.28.2.1, plperl kinda > works. But there's a lot of regression test failures, many of them > seemingly around error trapping. Here's a CI run testing various strawberry perl versions on windows. I did apply Victor's patch to make things at least compile on newer versions of perl. https://cirrus-ci.com/build/6290387791773696 - 5.32.1.1: fails with "src/pl/plperl/Util.c: loadable library and perl binaries are mismatched (got handshake key 0000000012800080,needed 0000000012900080)" - 5.30.3.1: crashes in plperl_trusted_init(), see "cat_dumps" step for backtrace - 5.28.2.1: doesn't crash, but lots of things don't seem to work, particularly around error handling (to see regression diff, click on regress_diffs near the top, and navigate to src/pl/plperl) - 5.24.4.1 and 5.26.3.1: pass The 5.32.1.1 issue looks like it might actually a problem in strawberry perl perhaps? But the rest not so much. Greetings, Andres Freund
Hi, On 2021-10-04 14:38:16 -0700, Andres Freund wrote: > 3) When building against strawberry perl 5.32.1.1 I see errors when loading > plperl The error is: loadable library and perl binaries are mismatched (got handshake key 0000000012800080, needed 0000000012900080) A bunch of research led me to believe this is because the struct sizes of PerlInterpreter differ between perl being compiled and us embedding perl. After a lot of headscratching [1], I got a struct layout of both a gcc compiled (just a test.c including the relevant headers) and and the msvc compiled plperl.dll. And indeed they differ: msvc: +0x42d Iin_utf8_COLLATE_locale : Bool +0x42e Iin_utf8_turkic_locale : Bool +0x42f Ilocale_utf8ness : [256] Char +0x530 Iwarn_locale : Ptr64 sv +0x538 Icolors : [6] Ptr64 Char +0x568 Ipeepp : Ptr64 void .. +0x1278 IPrivate_Use : Ptr64 sv gcc: /* 0x042d | 0x0001 */ _Bool Iin_utf8_COLLATE_locale; /* 0x042e | 0x0001 */ _Bool Iin_utf8_turkic_locale; /* 0x0430 | 0x0004 */ int Ilc_numeric_mutex_depth; /* 0x0434 | 0x0100 */ char Ilocale_utf8ness[256]; /* 0x0538 | 0x0008 */ SV *Iwarn_locale; /* 0x0540 | 0x0030 */ char *Icolors[6]; /* 0x0570 | 0x0008 */ peep_t Ipeepp; ... /* 0x1280 | 0x0008 */ SV *IPrivate_Use; The gcc version has a Ilc_numeric_mutex_depth that the msvc version doesn't. The relevant part of intrpvar.h: PERLVAR(I, in_utf8_turkic_locale, bool) #if defined(USE_ITHREADS) && ! defined(USE_THREAD_SAFE_LOCALE) PERLVARI(I, lc_numeric_mutex_depth, int, 0) /* Emulate general semaphore */ #endif PERLVARA(I, locale_utf8ness, 256, char) This conditional piece didn't yet exist in 5.26.n. Which is why that's the last version that actually works. USE_ITHREADS is defined in perls' config.h, but USE_THREAD_SAFE_LOCALE is derived from some other stuff. So that's the culprit. I gotta do something else for a bit, so I'll stop here for now. The error message about mismatched lib / perl binary could really use a bit more detail. It's pretty darn annoying to figure out right now what it could mean. Greetings, Andres Freund [1] On linux I'd just use pahole to display struct layouts, but on windows... Neither of the windows perl installations comes with debug symbols, afaict. For the gcc definition: I compiled a test.c with msys ucrt64, including -g3, set a breakpoint on main, dumped the struct with "ptype /ox my_interp" For the msvc definition: connected cdb.exe to a live backend, did a CREATE EXTENSION plperl, cdb stopped at exit, then I could dump the type with "dt plperl!PerlInterpreter" I'm sure there's a better way. And of course I'm mainly including this for a future self that might remember needing to something like this before...
Hi,
On 2022-01-30 12:56:16 -0800, Andres Freund wrote:
> The gcc version has a Ilc_numeric_mutex_depth that the msvc version
> doesn't. The relevant part of intrpvar.h:
>
> PERLVAR(I, in_utf8_turkic_locale, bool)
> #if defined(USE_ITHREADS) && ! defined(USE_THREAD_SAFE_LOCALE)
> PERLVARI(I, lc_numeric_mutex_depth, int, 0) /* Emulate general semaphore */
> #endif
> PERLVARA(I, locale_utf8ness, 256, char)
>
> This conditional piece didn't yet exist in 5.26.n. Which is why that's the
> last version that actually works.
>
> USE_ITHREADS is defined in perls' config.h, but USE_THREAD_SAFE_LOCALE is
> derived from some other stuff. So that's the culprit.
>
>
> I gotta do something else for a bit, so I'll stop here for now.
The difference originates in this bit in plperl.h:
/* XXX The next few defines are unfortunately duplicated in makedef.pl, and
* changes here MUST also be made there */
# if ! defined(HAS_SETLOCALE) && defined(HAS_POSIX_2008_LOCALE)
# define USE_POSIX_2008_LOCALE
# ifndef USE_THREAD_SAFE_LOCALE
# define USE_THREAD_SAFE_LOCALE
# endif
/* If compiled with
* -DUSE_THREAD_SAFE_LOCALE, will do so even
* on unthreaded builds */
# elif (defined(USE_ITHREADS) || defined(USE_THREAD_SAFE_LOCALE)) \
&& ( defined(HAS_POSIX_2008_LOCALE) \
|| (defined(WIN32) && defined(_MSC_VER) && _MSC_VER >= 1400)) \
&& ! defined(NO_THREAD_SAFE_LOCALE)
# ifndef USE_THREAD_SAFE_LOCALE
# define USE_THREAD_SAFE_LOCALE
# endif
# ifdef HAS_POSIX_2008_LOCALE
# define USE_POSIX_2008_LOCALE
# endif
# endif
#endif
Specifically where USE_THREAD_SAFE_LOCALE is defined for msvc. Which explains
why the same perl build ends up with different definitions for
PerlInterpreter, depending on headers getting compiled with gcc or
msvc.
Seems pretty clear that this is something that should be determined at build,
rather than at #include time?
I tested that just forcing the msvc build to behave the same using
NO_THREAD_SAFE_LOCALE makes the tests pass. Yay. But it's obviously not a
great solution - I'm not aware of a windows perl distribution that uses msvc,
but who knows.
> The error message about mismatched lib / perl binary could really use a bit
> more detail. It's pretty darn annoying to figure out right now what it could
> mean.
I wonder if we could do something to improve that on our side. This isn't the
first time we've hunted down this kind of mismatch. It'd be much friendlier if
we could get an error at build time, rather than runtime.
Greetings,
Andres Freund
On Sun, Jan 30, 2022 at 02:16:59PM -0800, Andres Freund wrote: > Specifically where USE_THREAD_SAFE_LOCALE is defined for msvc. Which explains > why the same perl build ends up with different definitions for > PerlInterpreter, depending on headers getting compiled with gcc or > msvc. > > Seems pretty clear that this is something that should be determined at build, > rather than at #include time? Agreed. > I tested that just forcing the msvc build to behave the same using > NO_THREAD_SAFE_LOCALE makes the tests pass. Yay. But it's obviously not a > great solution - I'm not aware of a windows perl distribution that uses msvc, > but who knows. Last I looked (~2017), EDB distributed an MSVC-built Perl as the designated Perl to use with https://www.postgresql.org/download/windows/ plperl. > > The error message about mismatched lib / perl binary could really use a bit > > more detail. It's pretty darn annoying to figure out right now what it could > > mean. > > I wonder if we could do something to improve that on our side. This isn't the > first time we've hunted down this kind of mismatch. It'd be much friendlier if > we could get an error at build time, rather than runtime. The MSVC build system does give a build-time error ("Perl test fails with or without ...") for a Perl ABI mismatch. It would be a simple matter of programming to have the configure+gmake build system do the same.
Hi, On 2022-01-30 15:14:32 -0800, Noah Misch wrote: > Last I looked (~2017), EDB distributed an MSVC-built Perl as the designated > Perl to use with https://www.postgresql.org/download/windows/ plperl. Ah, interesting. I didn't find a perl binary in the archive offered, and I didn't immediately figure out how to extract the files from the installer, so I didn't check further. > > > The error message about mismatched lib / perl binary could really use a bit > > > more detail. It's pretty darn annoying to figure out right now what it could > > > mean. > > > > I wonder if we could do something to improve that on our side. This isn't the > > first time we've hunted down this kind of mismatch. It'd be much friendlier if > > we could get an error at build time, rather than runtime. > > The MSVC build system does give a build-time error ("Perl test fails with or > without ...") for a Perl ABI mismatch. Hm? I encountered this on an msvc build, building against strawberry perl (and then also against msys ucrt perl, I was trying to exclude a problem in strawberry perl). So perl is gcc built and postgres with msvc. It fails when creating the plperl extension, with loadable library and binaries are mismatched (got handshake key 0000000012800080, needed 0000000012900080) but not at build time. Ah, I see. The problem is that the test is only done for 32bit perl. I guess this stuff would need to be extracted in a helper function, so we can use it for different defines without a lot of repetition. Greetings, Andres Freund
On 10/4/21 18:02, Andres Freund wrote:
> Hi,
>
> On 2021-10-04 14:38:16 -0700, Andres Freund wrote:
>> 2) For some reason src/tools/install.pl doesn't install plperl[u].control,
>> plperl[u]--1.0.sql - But apparently the buildfarm doesn't have that issue,
>> because drongo successfully ran the plperl tests?
> Oh, figured that one out: Install.pm checks the current directory for
> config.pl - but my invocation was from the source tree root (which is
> supported for most things). Because of that it skipped installing plperl, as
> it though it wasn't installed.
We should fix that, maybe along these lines?
iff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm
index 8de79c618c..75e91f73b3 100644
--- a/src/tools/msvc/Install.pm
+++ b/src/tools/msvc/Install.pm
@@ -59,6 +59,8 @@ sub Install
our $config = shift;
unless ($config)
{
+ # we expect config.pl and config_default.pl to be here
+ chdir 'src/tools/msvc' if -d 'src/tools/msvc';
# suppress warning about harmless redeclaration of $config
no warnings 'misc';
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On 2022-01-31 10:43:31 -0500, Andrew Dunstan wrote: > We should fix that, maybe along these lines? WFM.