Обсуждение: Other Win32 TODO items?
Hi all,
Following Andrew Dunstan's recent post, thought I'd mention a few other
things which I think will need to be considered for the Win32 port. I've run
into these in trying to get a single-process, multi-threaded version of
postgres to run on Win32. Clearly, if core developers concur, I don't mind
putting my hand up to provide patches for some or all of them... but I'll
need advice from core for (at least) the last two.
In rough order of annoyance:
* ioctlsocket_ret
- is not initialized to 1 (at least in the WIN_32 code I started
with!)
- ok, so big deal!
* select(0,NULL,NULL,NULL,&delay) doesn't, under Win32
- replace with Sleep for win32?
* Directory "slashes"
- still quite a large number of forward slashes all over the place
* stat() + extension differences
- execute bits returned by stat are based on file extension under
Win32
- therefore, binaries'll need to have .exe extension (ie. no symlink
for postmaster), and the calls to stat'll have to append ".exe".
* readdir()
- sets errno on "final" call (at least in mingw 3.1)
- mingw source code appears to have been corrected in latest
revision (1.4) of mingw / runtime / mingwex / dirent.c
- currently gets things like SlruScanDirectory and MoveOfflineLogs
really unhappy, but can be dealt with till the next mingw is available
* pipe() replacement
- call is available under Win32, and returns "file" handle
- however, cannot select() on a file handle under Win32 (only socket
handles)
- this is a problem in code where postgres may be required to wait
on a pipe handle and a socket simultaneously (for instance,
pgstat_recvbuffer).
unlink() + rename()
- appears the "loop until unlinked" version of pgunlink can cause
infinite looping
- rename() has the same problem
- with my hacked version, I've had cases where unlink is invoked
which "loops" forever in pgunlink because an open connection still has that
file handle open (of course, this might be cause of something I've messed up
in my changes, but the little checking I've performed suggests that this is
probably not the case).
- FWIW, this prompted my "Dumb unlink question" of two days ago
(http://archives.postgresql.org/pgsql-hackers-win32/2003-10/msg00029.php),
which was so dumb that no one responded :-(
That's about the worst of it. For anyone interested, the multi-threaded
hacks I've done have produced a win32 postgres which is approaching the
state needed to be good enough for my work's internal purposes...
However, in doing so, it has become all too clear that, at least until MingW
supports the __declspec(thread) directive, this is not the approach that
ought to be taken.
Cheers,
Claudio
---
WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS:
THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041
TEL: +61 2 9555 1544 FAX: +61 2 9555 6911
Certain disclaimers and policies apply to all email sent from Memetrics.
For the full text of these disclaimers and policies see
http://www.memetrics.com/emailpolicy.html
----- Original Message ----- From: "Claudio Natoli" <claudio.natoli@memetrics.com> To: "pgsql-hackers-win32" <pgsql-hackers-win32@postgresql.org> Sent: Saturday, November 01, 2003 3:42 AM Subject: [pgsql-hackers-win32] Other Win32 TODO items? > > Hi all, > > Following Andrew Dunstan's recent post, thought I'd mention a few other > things which I think will need to be considered for the Win32 port. I've run > into these in trying to get a single-process, multi-threaded version of > postgres to run on Win32. Clearly, if core developers concur, I don't mind > putting my hand up to provide patches for some or all of them... but I'll > need advice from core for (at least) the last two. > > In rough order of annoyance: > > * ioctlsocket_ret > - is not initialized to 1 (at least in the WIN_32 code I started > with!) > - ok, so big deal! > > * select(0,NULL,NULL,NULL,&delay) doesn't, under Win32 > - replace with Sleep for win32? > > * Directory "slashes" > - still quite a large number of forward slashes all over the place > Win32 seems to be quite happy with forward slashes. That's why I canonicalised all paths to use them in initdb. I tested this using mingw shell, cygwin shell and XPHE standard command shell. > * stat() + extension differences > - execute bits returned by stat are based on file extension under > Win32 > - therefore, binaries'll need to have .exe extension (ie. no symlink > for postmaster), and the calls to stat'll have to append ".exe". > In initdb I didn't even do a stat check for executable bit on windows, as I already knew I was stat-ing a .exe. > * readdir() > - sets errno on "final" call (at least in mingw 3.1) > - mingw source code appears to have been corrected in latest > revision (1.4) of mingw / runtime / mingwex / dirent.c > - currently gets things like SlruScanDirectory and MoveOfflineLogs > really unhappy, but can be dealt with till the next mingw is available I had not noticed this (wasn't checking errno, just using code like while ((file = readdir(dir)) != NULL) > > * pipe() replacement > - call is available under Win32, and returns "file" handle > - however, cannot select() on a file handle under Win32 (only socket > handles) > - this is a problem in code where postgres may be required to wait > on a pipe handle and a socket simultaneously (for instance, > pgstat_recvbuffer). > ugghh! > unlink() + rename() > - appears the "loop until unlinked" version of pgunlink can cause > infinite looping > - rename() has the same problem > - with my hacked version, I've had cases where unlink is invoked > which "loops" forever in pgunlink because an open connection still has that > file handle open (of course, this might be cause of something I've messed up > in my changes, but the little checking I've performed suggests that this is > probably not the case). > - FWIW, this prompted my "Dumb unlink question" of two days ago > (http://archives.postgresql.org/pgsql-hackers-win32/2003-10/msg00029.php), > which was so dumb that no one responded :-( > This needs further diagnosis IMHO. > > That's about the worst of it. For anyone interested, the multi-threaded > hacks I've done have produced a win32 postgres which is approaching the > state needed to be good enough for my work's internal purposes... > > However, in doing so, it has become all too clear that, at least until MingW > supports the __declspec(thread) directive, this is not the approach that > ought to be taken. > Do we need to talk to the MinGW people about this? > Cheers, > Claudio > > --- > WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: > THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 > TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 (waves across american continent and pacific ocean in direction of homeland ;-) cheers andrew
> > * Directory "slashes" > > - still quite a large number of forward slashes all over the place > > > > Win32 seems to be quite happy with forward slashes. That's why I > canonicalised all paths to use them in initdb. I tested this > using mingw shell, cygwin shell and XPHE standard command shell. mingw + cygwin are certainly happy with forward slashes, but windows certainly isn't in general (I have no familiarity with XPHE, so I can't comment here). Let me put it another way: there are a bunch of places (mostly in sprintf's where a file name is being formatted) where I had to change to backslashes to stop postgres PANICking about not being able to find/open files [on Win2K at least]. > > * stat() + extension differences > > - execute bits returned by stat are based on file extension under > > Win32 > > - therefore, binaries'll need to have .exe extension (ie. no symlink > > for postmaster), and the calls to stat'll have to append ".exe". > > > > In initdb I didn't even do a stat check for executable bit on > windows, as I already knew I was stat-ing a .exe. Perhaps I should've been more specific. The ValidateBinary call (called by FindExec) is going to fail unless it gets passed a filename ending in ".exe", and taking these calls out of the call path for win32 is unlikely to be an acceptable solution. :-) In any case, this isn't a big deal... > > However, in doing so, it has become all too clear that, at least until MingW > > supports the __declspec(thread) directive, this is not the approach that > > ought to be taken. > > > Do we need to talk to the MinGW people about this? Already done. Turns out it is already being worked on. It won't make the gcc 3.3 release, but perhaps gcc 3.4. In any case, it doesn't really matter, as the approach currently advocated by core for win32 is a multi-process one. > (waves across american continent and pacific ocean in direction of homeland ;-) Hey there! Cheers, Claudio --- WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see http://www.memetrics.com/emailpolicy.html
Claudio Natoli wrote: >>>* Directory "slashes" >>>- still quite a large number of forward slashes all over the place >>> >>> >>> >>Win32 seems to be quite happy with forward slashes. That's why I >>canonicalised all paths to use them in initdb. I tested this >>using mingw shell, cygwin shell and XPHE standard command shell. >> >> > >mingw + cygwin are certainly happy with forward slashes, but windows >certainly isn't in general (I have no familiarity with XPHE, so I can't >comment here). > >Let me put it another way: there are a bunch of places (mostly in sprintf's >where a file name is being formatted) where I had to change to backslashes >to stop postgres PANICking about not being able to find/open files [on Win2K >at least]. > To the best of my knowledge, only the command processor cares about this, and you can get around it by quoting the command - see my initdb.c for examples. (this is also why I pass PGDATA via the environment and never via the command line - the Windows command processor is quite dumb about multiple quoted strings). AFAIK direct library calls like stat() or fopen() should accept forward slashes. It may be that libraries other than MSVCRT act differently, although that would be very odd, even for M$. What context are you using the formatted string in? > > > > >>>* stat() + extension differences >>>- execute bits returned by stat are based on file extension under >>>Win32 >>>- therefore, binaries'll need to have .exe extension (ie. no symlink >>>for postmaster), and the calls to stat'll have to append ".exe". >>> >>> >>> >>In initdb I didn't even do a stat check for executable bit on >>windows, as I already knew I was stat-ing a .exe. >> >> > >Perhaps I should've been more specific. The ValidateBinary call (called by >FindExec) is going to fail unless it gets passed a filename ending in >".exe", and taking these calls out of the call path for win32 is unlikely to >be an acceptable solution. :-) In any case, this isn't a big deal... > > > Regardless, you can only stat a file, and windows executables have .exe extensions. Of course we have to program for this (again. see initdb.c for examples). > > >>>However, in doing so, it has become all too clear that, at least until >>> >>> >MingW > > >>>supports the __declspec(thread) directive, this is not the approach that >>>ought to be taken. >>> >>> >>Do we need to talk to the MinGW people about this? >> >> > >Already done. Turns out it is already being worked on. It won't make the gcc >3.3 release, but perhaps gcc 3.4. In any case, it doesn't really matter, as >the approach currently advocated by core for win32 is a multi-process one. > > > For now. I suspect that threading will be wanted before too long. cheers andrew > >
> To the best of my knowledge, only the command processor cares about > this, and you can get around it by quoting the command - see > my initdb.c for examples. (this is also why I pass PGDATA via the environment and > never via the command line - the Windows command processor is > quite dumb about multiple quoted strings). AFAIK direct library calls > like stat() or fopen() should accept forward slashes. It may be that > libraries other than MSVCRT act differently, although that would be very odd, > even for M$. They will, as long as there isn't a mix of forward and backward slashes, which is currently the case in the backend code. Either canonicalize on forward, or #define out backslashes in Win32. In either case, it is an outstanding TODO item for postgres under win32. > For now. I suspect that threading will be wanted before too long. Threading'll bring in a bunch of other items (for instance, file handles + memory leaks that currently are closed/recovered when the backend process dies), but one step at a time :-) Cheers, Claudio --- WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS: THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041 TEL: +61 2 9555 1544 FAX: +61 2 9555 6911 Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see http://www.memetrics.com/emailpolicy.html
Claudio Natoli wrote: >>To the best of my knowledge, only the command processor cares about >>this, and you can get around it by quoting the command - see >>my initdb.c for examples. (this is also why I pass PGDATA via the >> >> >environment and > > >>never via the command line - the Windows command processor is >>quite dumb about multiple quoted strings). AFAIK direct library calls >>like stat() or fopen() should accept forward slashes. It may be that >>libraries other than MSVCRT act differently, although that would be very >> >> >odd, > > >>even for M$. >> >> > >They will, as long as there isn't a mix of forward and backward slashes, >which is currently the case in the backend code. Either canonicalize on >forward, or #define out backslashes in Win32. In either case, it is an >outstanding TODO item for postgres under win32. > > > Right. FWIW, I found canonicalization by far the simpler approach - which I adopted after tying myself in knots doing things the other way. cheers andrew
Hello, I have arranged to have 1 full time dedicated programmer to the Win32 port. I can commit this person for 320 hours. I may also be able to add an additional full time programmer (for a total of 2) to the win32 project in a short period of time. However before I commit the resources I would like to get an idea of the direction the project is going. Specifically, I believe and through conversations with several Win32 programmers I know, that using create process is a bad idea and that we should move to a threaded model (at least for Win32). What are people's thoughts on this. Is there a development plan for the Win32 version that I perhaps haven't seen or are we just kind of hacking it as we go along? Sincerely, Joshua D. Drake -- Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC Postgresql support, programming shared hosting and dedicated hosting. +1-503-222-2783 - jd@commandprompt.com - http://www.commandprompt.com Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org
Claudio Natoli wrote: > > Hi all, > > Following Andrew Dunstan's recent post, thought I'd mention a few other > things which I think will need to be considered for the Win32 port. I've run > into these in trying to get a single-process, multi-threaded version of > postgres to run on Win32. Clearly, if core developers concur, I don't mind > putting my hand up to provide patches for some or all of them... but I'll > need advice from core for (at least) the last two. > > In rough order of annoyance: > > * ioctlsocket_ret > - is not initialized to 1 (at least in the WIN_32 code I started > with!) > - ok, so big deal! Fix just committed to postmaster.c. Thanks. I am splitting up these emails so it is easier for people to follow. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
Claudio Natoli wrote:
>
> Hi all,
>
> Following Andrew Dunstan's recent post, thought I'd mention a few other
> things which I think will need to be considered for the Win32 port. I've run
> into these in trying to get a single-process, multi-threaded version of
> postgres to run on Win32. Clearly, if core developers concur, I don't mind
> putting my hand up to provide patches for some or all of them... but I'll
> need advice from core for (at least) the last two.
>
> In rough order of annoyance:
>
> * ioctlsocket_ret
> - is not initialized to 1 (at least in the WIN_32 code I started
> with!)
> - ok, so big deal!
>
> * select(0,NULL,NULL,NULL,&delay) doesn't, under Win32
> - replace with Sleep for win32?
I have added this to the Win32 project page:
# Problems with select()
* Using as timer does't work, select(NULL, , timeout), use Sleep()?
* Only works on sockets, not file descriptors as returned by pipe()
* Must properly deal with signals
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Andrew Dunstan wrote:
>
>
> Claudio Natoli wrote:
>
> >>To the best of my knowledge, only the command processor cares about
> >>this, and you can get around it by quoting the command - see
> >>my initdb.c for examples. (this is also why I pass PGDATA via the
> >>
> >>
> >environment and
> >
> >
> >>never via the command line - the Windows command processor is
> >>quite dumb about multiple quoted strings). AFAIK direct library calls
> >>like stat() or fopen() should accept forward slashes. It may be that
> >>libraries other than MSVCRT act differently, although that would be very
> >>
> >>
> >odd,
> >
> >
> >>even for M$.
> >>
> >>
> >
> >They will, as long as there isn't a mix of forward and backward slashes,
> >which is currently the case in the backend code. Either canonicalize on
> >forward, or #define out backslashes in Win32. In either case, it is an
> >outstanding TODO item for postgres under win32.
> >
> >
> >
>
> Right. FWIW, I found canonicalization by far the simpler approach -
> which I adopted after tying myself in knots doing things the other way.
Added to Win32 project page:
Consistenly use forward slashes in paths, convert backslashes to
forward slashes
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Claudio Natoli wrote: > > Hi all, > > Following Andrew Dunstan's recent post, thought I'd mention a few other > things which I think will need to be considered for the Win32 port. I've run > into these in trying to get a single-process, multi-threaded version of > postgres to run on Win32. Clearly, if core developers concur, I don't mind > putting my hand up to provide patches for some or all of them... but I'll > need advice from core for (at least) the last two. > > In rough order of annoyance: > > * ioctlsocket_ret > - is not initialized to 1 (at least in the WIN_32 code I started > with!) > - ok, so big deal! Fixed. > * select(0,NULL,NULL,NULL,&delay) doesn't, under Win32 > - replace with Sleep for win32? On TODO. > * Directory "slashes" > - still quite a large number of forward slashes all over the place On TODO. > * stat() + extension differences > - execute bits returned by stat are based on file extension under > Win32 > - therefore, binaries'll need to have .exe extension (ie. no symlink > for postmaster), and the calls to stat'll have to append ".exe". Fixed. > * readdir() > - sets errno on "final" call (at least in mingw 3.1) > - mingw source code appears to have been corrected in latest > revision (1.4) of mingw / runtime / mingwex / dirent.c > - currently gets things like SlruScanDirectory and MoveOfflineLogs > really unhappy, but can be dealt with till the next mingw is available We will just use the updated MinGW when it is released. Added to TODO. > * pipe() replacement > - call is available under Win32, and returns "file" handle > - however, cannot select() on a file handle under Win32 (only socket > handles) > - this is a problem in code where postgres may be required to wait > on a pipe handle and a socket simultaneously (for instance, > pgstat_recvbuffer). Added to TODO. > unlink() + rename() > - appears the "loop until unlinked" version of pgunlink can cause > infinite looping > - rename() has the same problem > - with my hacked version, I've had cases where unlink is invoked > which "loops" forever in pgunlink because an open connection still has that > file handle open (of course, this might be cause of something I've messed up > in my changes, but the little checking I've performed suggests that this is > probably not the case). > - FWIW, this prompted my "Dumb unlink question" of two days ago > (http://archives.postgresql.org/pgsql-hackers-win32/2003-10/msg00029.php), > which was so dumb that no one responded :-( Added to TODO. > That's about the worst of it. For anyone interested, the multi-threaded > hacks I've done have produced a win32 postgres which is approaching the > state needed to be good enough for my work's internal purposes... > > However, in doing so, it has become all too clear that, at least until MingW > supports the __declspec(thread) directive, this is not the approach that > ought to be taken. Agreed. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
Joshua D. Drake wrote:
> Hello,
>
> I have arranged to have 1 full time dedicated programmer to the Win32
> port. I
> can commit this person for 320 hours. I may also be able to add an
> additional
> full time programmer (for a total of 2) to the win32 project in a short
> period of time.
> However before I commit the resources I would like to get an idea of the
> direction
> the project is going.
>
> Specifically, I believe and through conversations with several Win32
> programmers
> I know, that using create process is a bad idea and that we should move to
> a threaded model (at least for Win32).
>
> What are people's thoughts on this. Is there a development plan for
> the Win32
> version that I perhaps haven't seen or are we just kind of hacking it as
> we go along?
Basically, I think we are going in the right direction on the Win32
port:
http://momjian.postgresql.org/main/writings/pgsql/win32.html
If we need threading in the future, we can add it but the basic work
still needs to be done, and I think CreateProcess will be a minor issue.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073