Обсуждение: Custom C function - is palloc broken?

First off, I am developing custom c functions for PostgreSQL 8.3 in
Windows (sorry... not my choice). It appears that there is some
underlying problem in the server that is causing a runtime error to
crash the server when I use palloc. For example... I run the
following:

PG_FUNCTION_INFO_V1(add_one);
Datum
add_one(PG_FUNCTION_ARGS)
{
    int32        arg = PG_GETARG_INT32(0);
    PG_RETURN_INT32(arg + 1);
}

and it works just fine. NOTE: this is directly from the tutorial
directory. Now, when I add a palloc statement in there it crashes:


PG_FUNCTION_INFO_V1(add_one);
Datum
add_one(PG_FUNCTION_ARGS)
{
    int32        arg = PG_GETARG_INT32(0);
    palloc(sizeof(Point));
    PG_RETURN_INT32(arg + 1);
}

Running "SELECT add_one(1);" crashes the server. The logs show this:

2008-05-03 21:29:08 MDT LOG:  server process (PID 3008) was terminated
by exception 0xC0000005
2008-05-03 21:29:08 MDT HINT:  See C include file "ntstatus.h" for a
description of the hexadecimal value.
2008-05-03 21:29:08 MDT LOG:  terminating any other active server processes
2008-05-03 21:29:08 MDT LOG:  all server processes terminated; reinitializing
2008-05-03 21:29:09 MDT FATAL:  pre-existing shared memory block is still in use
2008-05-03 21:29:09 MDT HINT:  Check if there are any old server
processes still running, and terminate them.

I also get the exact same error when I try to run either the copytext
or concat_text functions from the same funcs_new.c file in the
tutorial directory. This essentially means that I cannot write any
UDFs that require memory allocation or text parameters. Heron seems to
be experiencing the same thing. Is this a bug in 8.3? Can anyone help?

Nathan Thatcher wrote:
> First off, I am developing custom c functions for PostgreSQL 8.3 in
> Windows (sorry... not my choice). It appears that there is some
> underlying problem in the server that is causing a runtime error to
> crash the server when I use palloc.

[Assuming you're using the stock 8.3.1 binaries and MSVC++8.0 or the
Express Edition of the same]:

Are you 100% certain you are using the same compiler and version that
PostgreSQL was built with, and that you have linked to the *exact* same
runtime library that the backend was linked to? These issues seem to be
a major cause of all sorts of fun problems especially for apps that
originated in the happier, saner UNIX world.

PostgreSQL probably doesn't put in the absurd effort required to avoid
freeing memory in a different DLL to where it was allocated [*]. Having
origins in nice sane UNIX environments where everybody just uses the
same compiler and runtime library it probably does not, this is quite
understandable. However, if I'm right about that (I have not verified
it) then you MUST link to the exact same runtime library that the
backend was linked to or things WILL break. Using the same version of
the same compiler would also be a good idea.

You'll also have to use a dynamic version of the runtime. Linking to a
static version will not work properly, because each DLL will have its
own private memory allocator.

Use Dependency Walker (depends.exe, from http://dependencywalker.com/)
to examine the names and resolved paths of the linked libraries. If you
are linking to a different runtime to the backend, or to multiple
runtimes, things are highly likely to explode.

The stock 8.3.1 win32 binaries appear to be linked to a copy of
MSVCR80.DLL in the WinSxS library tree. That means that you must link
only to MSCVR80.DLL (via the import library msvcrt.lib), which is the
multi-threaded non-debug dynamic runtime for Visual C++ 8. Make sure
you're using the same copy in the same WinSxS location, not a subtly
different version (say, from a VC++8.0 beta).

If your library/extension is linked to any other runtimes, like
msvcr80d.dll or msvcm80.dll, things will probably break.

If your library links to LIBCMT.LIB or LIBCMTD.LIB (note: these may not
appear in dependency walker as they're static libraries; check your
linker command line) things will CERTAINLY break.

That goes for MinGW, too. Do not expect a shared library compiled by
MinGW to load and run happily in an executable built with MSVC++ unless
all the public interfaces are very careful about how they handle memory
allocation, ownership and deallocation. Either build Pg with MinGW too
(if that's supported or even possible) or preferably just get the Visual
C++ Express Edition 8.0 from MS and use that to build your extension.

BTW: Rather than specifying the import library for the C library
directly as a linker argument, use the /MD flag in the linker options or
set it in the project file. If you are not using VC++ 8.0 it's not as
simple as just using /MD ; you might be able to fudge it with
/NODEFAULTLIB but I really do not know.

See:
     http://msdn.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx
for some information on the surprising variety of runtime libraries
provided by MSVC++ 8.0 alone.


 > I also get the exact same error when I try to run either the copytext
 > or concat_text functions from the same funcs_new.c file in the
 > tutorial directory. This essentially means that I cannot write any
 > UDFs that require memory allocation or text parameters. Heron seems to
 > be experiencing the same thing. Is this a bug in 8.3? Can anyone help?

Most text handling involves memory allocations and deallocations. I
would not be at all surprised if your problem was mismatched compilers
or runtimes.



* Yes, there's more to it than just avoiding freeing memory in a
different DLL than it was allocated in. File descriptors, for example,
need similar treatment, as do all sorts of other weird little corners
that aren't worth enumerating.

--
Craig Ringer

Nathan Thatcher wrote:
> First off, I am developing custom c functions for PostgreSQL 8.3 in
> Windows (sorry... not my choice). It appears that there is some
> underlying problem in the server that is causing a runtime error to
> crash the server when I use palloc. For example... I run the
> following:
>
Sorry, I forgot to to ask in my previous post:

Can you send me a complete, compileable example that illustrates the
problem, along with instructions on how to reproduce it? If possible,
send a compiled binary and any dependencies too. I'm curious to see if
this is in fact shared library issues.

--
Craig Ringer

Craig Ringer wrote:
 > Can you send me a complete, compileable example that illustrates the
 > problem, along with instructions on how to reproduce it? If possible,
 > send a compiled binary and any dependencies too. I'm curious to see if
 > this is in fact shared library issues.

I'll send you a .zip file separately with the actual code, but here are
the steps I took:

- Install Postgres 8.3.1
- Install MinGW 5.1.4
- Install the gettext snapshot from MinGW's downloads page under "snapshots"

Then, unzip the zip file I'll send you.  Make sure MinGW's bin directory
is in your windows PATH environment variable.  In a command prompt, go
to your unzipped directory and run mingw32-make.  The makefile I've
included will put the .dll in Postgres' lib/plugins directory (assuming
you installed postgres in its default location - modify the makefile if
that is not the case).

Once the .dll is ready, I created the function in Postgres using the
CREATE FUNCTION command given in the documentation, modified slightly:

CREATE FUNCTION copytext(text) RETURNS text
      AS '$libdir/plugins/mylib.dll', 'copytext'
      LANGUAGE C STRICT;

Then I used the following SELECT statement:

SELECT copytext(colname) FROM tablename;

where "colname" is a text column in table "tablename".  This statement
crashes the database server.

I read somewhere on Postgres' website that the win32 binaries were built
with MinGW - and in fact they must be, since MSVC++ does not have some
of the unix headers needed to build Postgres.

Your theory could very well be true - especially if the issue is with a
different version of gettext.  MinGW's website's snapshot is from 2006
or something, so it wouldn't surprise me if Postgres uses a newer version.

A zip file is attached to a separate e-mail to you.  Anyone else who
would like a copy may have one, just let me know, but it's basically
just the copytext example from the tutorials pasted into a .c file, and
a makefile.

- Dan

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
>  > Can you send me a complete, compileable example that illustrates
>  > the problem, along with instructions on how to reproduce it? If
>  > possible, send a compiled binary and any dependencies too. I'm
>  > curious to see if this is in fact shared library issues.
>
> I'll send you a .zip file separately with the actual code, but here
> are the steps I took:
>
> - Install Postgres 8.3.1
> - Install MinGW 5.1.4
> - Install the gettext snapshot from MinGW's downloads page under
> "snapshots"

This may be your problem. We have previously seen a lot of issues with
different versions of gettext on Windows. They do a lot of things
in ways that "you are not supposed to on Windows"...

Could you try a build without NLS?


> I read somewhere on Postgres' website that the win32 binaries were
> built with MinGW - and in fact they must be, since MSVC++ does not
> have some of the unix headers needed to build Postgres.

No, as of 8.3 they are build with MSVC.


//Magnus

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
>  > Can you send me a complete, compileable example that illustrates the
>  > problem, along with instructions on how to reproduce it? If possible,
>  > send a compiled binary and any dependencies too. I'm curious to see if
>  > this is in fact shared library issues.
>

> Then I used the following SELECT statement:
>
> SELECT copytext(colname) FROM tablename;
>
> where "colname" is a text column in table "tablename".  This statement
> crashes the database server.

I'm certainly seeing a server crash with your DLL.

I've built the example with Visual C++ 9 (I had a to make a couple of
changes, such as defining BUILDING_DLL and exporting the `copytext'
function with PGDLLIMPORT) and I get a backend crash with that too. Of
course, that's also using a different runtime, and it's far from
impossible that I'm doing something wrong in the build process.

I'm having some trouble getting the example building under Visual Studio
Express 2005 ( I had to install it to test, and it's *much* uglier than
2008 ) so I haven't been able to test it properly with a matching runtime.

Looking at Pg's headers, though, I'm going to say that my first guess
about mismatched runtimes is probably wrong. That's often the cause of
plugin/extension problems on win32, but I'm not sure I see what would be
causing a mismatched runtime related failure here. Not with memory
handling, anyway. palloc() is a macro, but it's implemented using
exported functions from the backend, so allocations are actually being
performed with the backend's runtime. No memory appears to be being
freed in the example code, only being handed over to the backend to be
released later.

I'll see if it works when built with VC++ Express 2005 if I can get it
to behave (ie if/when I figure out what stupid thing I've done). I'm not
too sure that'll actually be the issue though.

> Your theory could very well be true - especially if the issue is with a
> different version of gettext.  MinGW's website's snapshot is from 2006
> or something, so it wouldn't surprise me if Postgres uses a newer version.

I assume you needed gettext because PostgreSQL's headers expose Pg's use
of libintl to users of the Pg headers, so you needed libintl.h?

As the standard win32 build of Pg is built with gettext, but does not
ship the headers for the version of gettext used, it's a bit tricky to
build external C functions etc. It'd be nice to bundle those headers or
avoid including libintl.h and friends from Pg headers that're likely to
be needed for external C functions.

Ideally only .c sources would include libintl.h where required (but I
know how practical *that* sometimes turns out to be -
*cough*Freetype*cough*).

--
Craig Ringer

Magnus Hagander wrote:
> Dan "Heron" Myers wrote:

>> I read somewhere on Postgres' website that the win32 binaries were
>> built with MinGW - and in fact they must be, since MSVC++ does not
>> have some of the unix headers needed to build Postgres.
>
> No, as of 8.3 they are build with MSVC.

In case it's of any help, I was able to build the test with VC++ 8 and
confirmed that it does still crash at the palloc() call even when the
exact same runtime is used by both the test DLL and by postgres.exe
(according to depends.exe).

Anyway, the birds are chirping and I have to work way too soon...

--
Craig Ringer

So what options does that leave all of us who need to compile and run
our custom C functions in Windows?

On Sun, May 4, 2008 at 4:47 PM, Craig Ringer
<craig@postnewspapers.com.au> wrote:
> Magnus Hagander wrote:
>
>
> > Dan "Heron" Myers wrote:
> >
>
>
> >
> > > I read somewhere on Postgres' website that the win32 binaries were
> > > built with MinGW - and in fact they must be, since MSVC++ does not
> > > have some of the unix headers needed to build Postgres.
> > >
> >
> > No, as of 8.3 they are build with MSVC.
> >
>
>  In case it's of any help, I was able to build the test with VC++ 8 and
> confirmed that it does still crash at the palloc() call even when the exact
> same runtime is used by both the test DLL and by postgres.exe (according to
> depends.exe).
>
>  Anyway, the birds are chirping and I have to work way too soon...
>
>  --
>  Craig Ringer
>
>
>
>  --
>  Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
>  To make changes to your subscription:
>  http://www.postgresql.org/mailpref/pgsql-general
>

Nathan Thatcher wrote:
> So what options does that leave all of us who need to compile and run
> our custom C functions in Windows?

Debug further :-)

Because it's certainly possible to do it. IIRC, PostGIS for 8.3 is
build with MingW and it works fine. And other extensions are known
working when built with MSVC Express. So it should work in principle,
we just need to figure out exactly what the issue is in this specific
case.

I'm still thinking gettext could be the issue here.

//Magnus

Nathan Thatcher wrote:
> So what options does that leave all of us who need to compile and run
> our custom C functions in Windows?

After a bit of sleep and with the advantage of a now-working brain I've
got it working, at least with 8.3 .

The problem was with DLL linkage, especially the CurrentMemoryContext
global variable. I was defining BUILDING_DLL and using the PGDLLIMPORT
macro to export the copytext() function - but this was of course causing
everything in the Pg headers to be declared __declspec(dllexport)
instead of __declspec(dllimport) . AFAIK this is survivable for
functions, which will use a slower call thunk instead, but not for
exported variables.

As a result, MemoryContextAlloc(CurrentMemoryContext, (sz)) was being
called with a nonsensical pointer for CurrentMemoryContext and crashing.


To build the C function DLL correctly it correctly do not define the
BUILDING_DLL macro. Instead, define your own DLL export macros like:

#if defined(_MSC_VER) || defined(__MINGW32__)
#define COPYTEXT_EXPORT __declspec (dllexport)
#else
#define COPYTEXT_EXPORT
#endif

then declare your function as:


COPYTEXT_EXPORT Datum copytext2(PG_FUNCTION_ARGS) {
     // blah blah
}


It should now be exported in the DLL's interface correctly AND have
correct access to Pg's exported functions and variables.

I did notice that I'm getting some warnings about inconsistent DLL
linkage for Pg_magic_func and pg_finfo_copytext . These really need to
be using __declspec(dllexport) rather than using __declspec(dllimport)
via PGDLLIMPORT .


Maybe it's worth providing a PGMODULEEXPORT macro for PG_MODULE_MAGIC,
PG_FUNCTION_INFO_V1, and for module writers to use to export their
functions in the DLL interface. A module author would have to ensure
that BUILDING_MODULE was defined.

Here's how the PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1 macros would look:

/* This might want to go somewhere other than fmgr.h, like
  * pg_config_os.h alongside the definition of PGDLLIMPORT
  */
#if defined(_MSC_VER) || defined(__MINGW32__)
   #if defined(BUILDING_MODULE)
     #define PGMODULEEXPORT __declspec (dllexport)
   #else
     // Never actually used
     #define PGMODULEEXPORT __declspec (dllimport)
   #endif
#else
   #define PGMODULEEXPORT
#endif

#define PG_MODULE_MAGIC \
PGMODULEEXPORT  Pg_magic_struct * \
PG_MAGIC_FUNCTION_NAME(void) \
{ \
    static const Pg_magic_struct Pg_magic_data = PG_MODULE_MAGIC_DATA; \
    return &Pg_magic_data; \
} \
extern int no_such_variable

#define PG_FUNCTION_INFO_V1(funcname) \
PGMODULEEXPORT const Pg_finfo_record * \
CppConcat(pg_finfo_,funcname) (void) \
{ \
    static const Pg_finfo_record my_finfo = { 1 }; \
    return &my_finfo; \
} \
extern int no_such_variable



I've tested this definition and it produces a DLL that links correctly
and does so without the warnings of inconsistent DLL linkage produced by
the original versions (which declared the function __declspec(dllimport)
then defined it).

Anyway, the C function examples need some changes to work correctly on
win32. I've attached updated versions. These redefine the
PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1 macros, but if you omit that
code the module will still built, just with warnings, and will still work.

--
Craig Ringer
/*
 * PostgreSQL example C functions.
 *
 * This file must be built as a shared library or dll and
 * placed into the PostgreSQL `lib' directory. On Windows
 * it must link to postgres.lib .
 *
 * postgresql/include/server must be on your header search path.
 * With MSVC++ on win32 so must postgresql/include/server/port/win32_msvc .
 * With MinGW use postgresql/include/server/port/win32 .
 */

#if defined(_MSC_VER) || defined(__MINGW32__)
  #ifndef _USE_32BIT_TIME_T
    #define _USE_32BIT_TIME_T
  #endif
#endif

/* BUILDING_DLL causes the declarations in Pg's headers to be declared
 * __declspec(dllexport) which will break DLL linkage. */
#ifdef BUILDING_DLL
  #error Do not define BUILDING_DLL when building extension libraries
#endif

/* Ensure that Pg_module_function and friends are declared __declspec(dllexport) */
#ifndef BUILDING_MODULE
  #define BUILDING_MODULE
#endif

#include "postgres.h"
#include <string.h>
#include "fmgr.h"
#include "utils/geo_decls.h"

/*--------------- BEGIN REDEFINITION OF PG MACROS -------------------
 *
 * These rewritten versions of PG_MODULE_MAGIC and PG_FUNCTION_INFO_V1
 * declare the module functions as __declspec(dllexport) when building
 * a module. They also provide PGMODULEEXPORT for exporting functions
 * in user DLLs.
 */
#undef PG_MODULE_MAGIC
#undef PG_FUNCTION_INFO_V1

/* This might want to go somewhere other than fmgr.h, like
 * pg_config_os.h alongside the definition of PGDLLIMPORT
 */
#if defined(_MSC_VER) || defined(__MINGW32__)
  #if defined(BUILDING_MODULE)
    #define PGMODULEEXPORT __declspec (dllexport)
  #else
    // Never actually used
    #define PGMODULEEXPORT __declspec (dllimport)
  #endif
#else
  #define PGMODULEEXPORT
#endif

#define PG_MODULE_MAGIC \
PGMODULEEXPORT  Pg_magic_struct * \
PG_MAGIC_FUNCTION_NAME(void) \
{ \
    static const Pg_magic_struct Pg_magic_data = PG_MODULE_MAGIC_DATA; \
    return &Pg_magic_data; \
} \
extern int no_such_variable

#define PG_FUNCTION_INFO_V1(funcname) \
PGMODULEEXPORT const Pg_finfo_record * \
CppConcat(pg_finfo_,funcname) (void) \
{ \
    static const Pg_finfo_record my_finfo = { 1 }; \
    return &my_finfo; \
} \
extern int no_such_variable

/*--------------- END REDEFINITION OF PG MACROS -------------------*/


PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(add_one);

PGMODULEEXPORT Datum
add_one(PG_FUNCTION_ARGS)
{
    int32   arg = PG_GETARG_INT32(0);

    PG_RETURN_INT32(arg + 1);
}

/* by reference, fixed length */

PG_FUNCTION_INFO_V1(add_one_float8);

PGMODULEEXPORT Datum
add_one_float8(PG_FUNCTION_ARGS)
{
    /* The macros for FLOAT8 hide its pass-by-reference nature. */
    float8   arg = PG_GETARG_FLOAT8(0);

    PG_RETURN_FLOAT8(arg + 1.0);
}

PG_FUNCTION_INFO_V1(makepoint);

PGMODULEEXPORT Datum
makepoint(PG_FUNCTION_ARGS)
{
    /* Here, the pass-by-reference nature of Point is not hidden. */
    Point     *pointx = PG_GETARG_POINT_P(0);
    Point     *pointy = PG_GETARG_POINT_P(1);
    Point     *new_point = (Point *) palloc(sizeof(Point));

    new_point->x = pointx->x;
    new_point->y = pointy->y;

    PG_RETURN_POINT_P(new_point);
}

/* by reference, variable length */

PG_FUNCTION_INFO_V1(copytext);

PGMODULEEXPORT Datum
copytext(PG_FUNCTION_ARGS)
{
    text     *t = PG_GETARG_TEXT_P(0);
    /*
     * VARSIZE is the total size of the struct in bytes.
     */
    text     *new_t = (text *) palloc(VARSIZE(t));
    SET_VARSIZE(new_t, VARSIZE(t));
    /*
     * VARDATA is a pointer to the data region of the struct.
     */
    memcpy((void *) VARDATA(new_t), /* destination */
           (void *) VARDATA(t),     /* source */
           VARSIZE(t) - VARHDRSZ);  /* how many bytes */
    PG_RETURN_TEXT_P(new_t);
}

PG_FUNCTION_INFO_V1(concat_text);

PGMODULEEXPORT Datum
concat_text(PG_FUNCTION_ARGS)
{
    text  *arg1 = PG_GETARG_TEXT_P(0);
    text  *arg2 = PG_GETARG_TEXT_P(1);
    int32 new_text_size = VARSIZE(arg1) + VARSIZE(arg2) - VARHDRSZ;
    text *new_text = (text *) palloc(new_text_size);

    SET_VARSIZE(new_text, new_text_size);
    memcpy(VARDATA(new_text), VARDATA(arg1), VARSIZE(arg1) - VARHDRSZ);
    memcpy(VARDATA(new_text) + (VARSIZE(arg1) - VARHDRSZ),
           VARDATA(arg2), VARSIZE(arg2) - VARHDRSZ);
    PG_RETURN_TEXT_P(new_text);
}

Craig Ringer wrote:
> After a bit of sleep and with the advantage of a now-working brain I've
> got it working, at least with 8.3 .

Awesome, thanks :)

- Dan

Craig Ringer wrote:
> I've tested this definition and it produces a DLL that links correctly
> and does so without the warnings of inconsistent DLL linkage produced by
> the original versions (which declared the function __declspec(dllimport)
> then defined it).

Did you actually try to use those functions in a query, or did you just
build the DLL?

With the changes you suggested, I can still successfully build the DLL
(I could build it before, as well), but now EVERY function crashes
Postgres when I try to run it, instead of just the ones that use palloc.

Any ideas?

- Dan

Craig Ringer wrote:
> Which compiler did you use?

I've been using MinGW because I don't know how to turn off NLS to
compile with VC++.

- Dan

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
>> I've tested this definition and it produces a DLL that links correctly
>> and does so without the warnings of inconsistent DLL linkage produced
>> by the original versions (which declared the function
>> __declspec(dllimport) then defined it).
>
> Did you actually try to use those functions in a query, or did you just
> build the DLL?

I tested the functions, which all worked fine.

> With the changes you suggested, I can still successfully build the DLL
> (I could build it before, as well), but now EVERY function crashes
> Postgres when I try to run it, instead of just the ones that use palloc.

Which compiler did you use?

--
Craig Ringer

Craig Ringer wrote:
> I'm actually using a dummy header (just an empty file) for libintl.h .
> None of the involved module code actually uses anything from libintl,
> neither directly nor via any inlines/macros from postgresql's headers.
> I'm not even sure PostgreSQL needs to include the header in its own
> headers at all; it might be fine being included by the .c files. This
> shouldn't make any difference.

There are other files (such as netdb.h) which are also missing.  Do I
also need blank placeholder files for those to compile with VC++?  It
seems odd that they can compile PostgreSQL with Visual Studio (according
to Magnus, anyway) without several of the unix headers it depends on...
  and if they do, why don't they have some #ifdefs to check whether it's
building for windows and not include them if it is?

> - Is this with the original 8.3.1 binaries or custom ones built with
> MinGW and with NLS disabled?

The PostgreSQL binaries are the ones from the Windows installer.

> - Does it crash with the same error message & error code?

I don't get an error message per se...  The log file shows no errors,
but the query window I'm using (in pgadmin) loses its connection and
Visual Studio's just-in-time debugger pops up asking if I want to debug
postgres.exe... but if I say yes, it says it can't attach to the
process.  The crash is the same every time.

> - Are you testing with the source file I sent (that I know works) or
> with modified sources based on your original example?

I've tested both your file and mine, compiled with MinGW; both crash
every time I try any function.

- Dan

Dan "Heron" Myers wrote:
> Craig Ringer wrote:
>> I'm actually using a dummy header (just an empty file) for libintl.h .
>> None of the involved module code actually uses anything from libintl,
>> neither directly nor via any inlines/macros from postgresql's headers.
>> I'm not even sure PostgreSQL needs to include the header in its own
>> headers at all; it might be fine being included by the .c files. This
>> shouldn't make any difference.
>
> There are other files (such as netdb.h) which are also missing.  Do I
> also need blank placeholder files for those to compile with VC++?

Sorry, I forgot to mention those because I was lazy and just commented
them out in the Pg headers instead of doing the right thing and
providing a dummy include file. You do appear to need a dummy include,
unless I'm doing something wrong in my build setup. I don't see any
conditional inclusion of these headers, not any existing dummy headers
in the port/ directory, and there aren't any alternative copies of
`port.h' for use on win32, so I don't see how I could be doing it wrong.

Neither netdb.h nor pwd.h seem to actually be used (beyond being
included) on win32, at least with VC++

The other thing I had to do was add include guards on pg_config_os.h to
protect against repeat inclusion. VC++ 8.0 does not like the repeated
definition of `struct timezone' and `struct itimerval' (but 9.0 -
correctly - doesn't care). I have no idea how the standard binaries were
built with VC++ without that change - maybe there's a setting to
convince it to ignore the "problem" that's used in the project
file/makefile.

> It
> seems odd that they can compile PostgreSQL with Visual Studio (according
> to Magnus, anyway) without several of the unix headers it depends on...
>  and if they do, why don't they have some #ifdefs to check whether it's
> building for windows and not include them if it is?

It's entirely possible it was built with dummies for these headers,
perhaps from another project or provided by a win32 port of another
project. I don't know how the build environment for the official win32
binaries is set up.

It's more likely that I'm doing something stupid, but I don't at present
see what it's likely to be.

>> - Does it crash with the same error message & error code?
>
> I don't get an error message per se...  The log file shows no errors,
> but the query window I'm using (in pgadmin) loses its connection and
> Visual Studio's just-in-time debugger pops up asking if I want to debug
> postgres.exe... but if I say yes, it says it can't attach to the
> process.  The crash is the same every time.

And in the server log files (PG_INSTALL_DIR\data\pg_log) ?

>> - Are you testing with the source file I sent (that I know works) or
>> with modified sources based on your original example?
>
> I've tested both your file and mine, compiled with MinGW; both crash
> every time I try any function.

OK, so there are still issues at least with MinGW. I won't venture any
guesses as to what could be going on there; I can only suggest trying
with VC++ . I've done a test build with VC++ 9.0 (Express edition 2008)
as well and encountered no issues, however I would personally stick to
VC++ 8.0 for production use until the Pg binaries are built with 9.0 .

You're asking for trouble by mixing compilers, compiler versions, and
runtimes - even if it appears to work (which it does for VC 9, and not
for MinGW).

I'll see if I can find time to look into the mingw issues anyway, though
I don't know if I'm really up to the task of tracking those problems down.

I'd be interested in confirmation that you're able to get your code
working when you build it with VC++ .

--
Craig Ringer

"Dan \"Heron\" Myers" <heron@xnapid.com> writes:
> It seems odd that they can compile PostgreSQL with Visual Studio (according
> to Magnus, anyway)

And according to several buildfarm machines, as well as quite a few
other folks who have managed to build things successfully.  You need to
get over this concept of "Postgres is broken" and try to identify a
specific reason why your DLL builds are not working.

One thing you might try is looking at the build logs for the Windows
buildfarm members
http://www.pgbuildfarm.org/cgi-bin/show_status.pl
to see if you can spot what they're doing differently than you are.
Check out the contrib-module builds in particular, because I doubt there
are any of them that aren't doing palloc.

            regards, tom lane

Craig Ringer <craig@postnewspapers.com.au> writes:
> The other thing I had to do was add include guards on pg_config_os.h to
> protect against repeat inclusion. VC++ 8.0 does not like the repeated
> definition of `struct timezone' and `struct itimerval' (but 9.0 -
> correctly - doesn't care). I have no idea how the standard binaries were
> built with VC++ without that change

Maybe because it's only included once anyway?

I'm starting to wonder just how broken the tools you two are using
must be ... and to give thanks once again that I never got sucked
into trying to do development on Windows.

            regards, tom lane

Craig Ringer wrote:
> And in the server log files (PG_INSTALL_DIR\data\pg_log) ?

The log files contained no indication of an error, not even an "oops,
the server crashed" message.

> I'd be interested in confirmation that you're able to get your code
> working when you build it with VC++ .

I did get the examples.c file to work with VC++ 2008 Professional, so I
should be able to get the other functions I need to work.


Tom Lane wrote:
 > You need to
 > get over this concept of "Postgres is broken" and try to identify a
 > specific reason why your DLL builds are not working.

I don't think I ever said Postgres was broken.  I said Postgres was
crashing.  I expressed doubt that Postgres' win32 binary was built with
MSVC++... mostly because the website still says it was built with mingw.
  In any case, I was mostly confused that Postgres would build their
stuff with MSVC++ but not provide #ifndef WIN32 (or whatever) blocks
around the stuff that can be ignored in windows.

But that's all essentially irrelevant.  In the future I will keep my
speculation to myself lest I be accused of calling Postgres "broken".

 > I'm starting to wonder just how broken the tools you two are using
 > must be ...

If by "broken" you mean "a Windows-only compiler excluding unix-only
headers from its include directories", then I guess Visual Studio is
completely broken.  If by "broken" you mean "MinGW apparently can't be
used to compile .dlls for use with Postgres", then that is also correct.
  I don't think either set of tools is actually broken, though.

In any case, I did get the examples compiling with Visual C++ 2008
Professional, and the resulting .dll does work in Postgres.  If I run
into any other problems I'll be back.

Thanks,
Dan

Dan "Heron" Myers wrote:

> Tom Lane wrote:
>  > You need to
>  > get over this concept of "Postgres is broken" and try to identify a
>  > specific reason why your DLL builds are not working.
>
> I don't think I ever said Postgres was broken.  I said Postgres was
> crashing.  I expressed doubt that Postgres' win32 binary was built with
> MSVC++... mostly because the website still says it was built with mingw.

Just FYI, the "installer" project as of 8.3 (I don't know about previous
releases) is using MSVC++ as far as I know.

http://pginstaller.projects.postgresql.org/

I see now that the FAQ appears to be out of date. Magnus, Dave? Can we
get the Installer FAQ updated?

http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html


> Thanks,
> Dan

Sorry you are having so much trouble.

Sincerely,

Joshua D. Drake

Tom Lane wrote:
> Craig Ringer <craig@postnewspapers.com.au> writes:
>> The other thing I had to do was add include guards on pg_config_os.h to
>> protect against repeat inclusion. VC++ 8.0 does not like the repeated
>> definition of `struct timezone' and `struct itimerval' (but 9.0 -
>> correctly - doesn't care). I have no idea how the standard binaries were
>> built with VC++ without that change
>
> Maybe because it's only included once anyway?

It's included twice if WIN32 is not defined, but you're actually
building on win32. WIN32 is defined by all Microsoft build tools EXCEPT
Visual Studio Express Edition 2005's default win32 project, which is why
2005 wasn't working but 2008 was.

> I'm starting to wonder just how broken the tools you two are using
> must be ... and to give thanks once again that I never got sucked
> into trying to do development on Windows.

The tools are VERY broken if you're talking about VS Express Edition 2005.

My own Windows development is only reluctant, and only because it's more
hassle trying to walk someone through tracking down some insane issue
than to just test it myself. It's generally limited to doing occasional
test builds of software I'm involved with on windows to make sure it
still works. As a result, I'm even worse at Windows development than I
am on UNIX.

I avoid working on win32 where possible, because it's horrible. Doubly
so with Visual Studio 2005 express, which in its default install is
significantly different to what most win32 code expects.

As for "postgres is broken" - I'm not saying that at all. I do think
there's a problem with the module magic macros, but it won't stop
anything from working (though it might slow down function calls to user
modules). I do think `libintl.h' shouldn't be exposed in the public
headers if possible - but again, it won't break anything, it's just a
cosmetic/usability thing.

I've repeatedly noted that it's highly likely I'm doing something wrong
in my build setup.

Well, I was. As well as the issue with the WIN32 macro noted above, I
had omitted postgresql/include/server/port/win32 from my header search
path (thinking that postgresql/include/server/port/win32_msvc was a
replacement, not a supplement). This is what I get for working on these
things at stupid-o-clock in the morning.

So:

If using Visual Studio Express Edition 2005, make sure to set WIN32 in
the preprocessor definitions (or modify the default project file to set
them automatically).

The appropriate header search path is:

include/server
include/server/port/win32_msvc
include/server/port/win32

... and it appears that the examples in the documentation will not work
for win32/msvc without a __declspec(dllexport) attribute to ensure that
functions are exported, but otherwise work just fine if your build
environment is set up correctly.

--
Craig Ringer

On Tue, May 6, 2008 at 5:02 AM, Craig Ringer
<craig@postnewspapers.com.au> wrote:
> WIN32 is defined by all Microsoft build tools EXCEPT Visual Studio
> Express Edition 2005's default win32 project, which is why 2005 wasn't
> working but 2008 was.

It certainly is defined on my copy.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Tue, May 6, 2008 at 4:18 AM, Joshua D. Drake <jd@commandprompt.com> wrote:

> Just FYI, the "installer" project as of 8.3 (I don't know about previous
> releases) is using MSVC++ as far as I know.

It is. Visual Studio 2005 Professional w/SP1 to be precise.

> I see now that the FAQ appears to be out of date. Magnus, Dave? Can we get
> the Installer FAQ updated?
>
> http://pginstaller.projects.postgresql.org/faq/FAQ_windows.html

Will do.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com