Обсуждение: improved parallel make support
I have worked on some improvements on how we handle recursive make in our makefiles. Most places uses for loops, which has some disadvantages: parallel make doesn't work across directories, make -k doesn't work, and make -q doesn't work. Instead, I went with the approach that we already use in the src/backend directory, where we call the subordinate makes as target prerequisites. Note that because with this, parallel make really works, the rule dependencies must be correct. This has always been the case, but now it really shows up. A frequent issue is that this sort of thing no longer works: all: submake-libpgport zic zic: $(ZICOBJS) because this relies on the "all" target to execute its prerequisites in order. Instead, you need to write it like this: all: zic zic: $(ZICOBJS) | submake-libpgport (The bar is necessary so that zic isn't considered constantly out of date because it depends on a phony target.) This patch requires GNU make 3.80, because of the above "|" feature and the $(eval) function. Version 3.80 is dated October 2002, so it should be no problem, but I do occasionally read of make 3.79 around here; maybe it's time to get rid of that. I did put in a check that makes the build fail right away if a wrong version of make is used.
Вложения
Peter Eisentraut <peter_e@gmx.net> writes:
> This patch requires GNU make 3.80, because of the above "|" feature and
> the $(eval) function. Version 3.80 is dated October 2002, so it should
> be no problem, but I do occasionally read of make 3.79 around here;
> maybe it's time to get rid of that. I did put in a check that makes the
> build fail right away if a wrong version of make is used.
Do we have a handle on how many buildfarm members this will break?
(fwiw, my hpux box is running 3.79.1)
regards, tom lane
On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > This patch requires GNU make 3.80, because of the above "|" feature and > > the $(eval) function. Version 3.80 is dated October 2002, so it should > > be no problem, but I do occasionally read of make 3.79 around here; > > maybe it's time to get rid of that. I did put in a check that makes the > > build fail right away if a wrong version of make is used. > > Do we have a handle on how many buildfarm members this will break? I suppose we don't. One way to find out would be to commit just this bit +# We need the $(eval) function, which is available in GNU make 3.80. +# That also happens to be the version where the .VARIABLES variable +# was introduced, so this is a simple check. +ifndef .VARIABLES +$(error GNU make 3.80 or newer is required) +endif with a $(warning) instead, and let it run for a bit.
Peter Eisentraut <peter_e@gmx.net> writes:
> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
>> Do we have a handle on how many buildfarm members this will break?
> I suppose we don't. One way to find out would be to commit just this
> bit
> +# We need the $(eval) function, which is available in GNU make 3.80.
> +# That also happens to be the version where the .VARIABLES variable
> +# was introduced, so this is a simple check.
> +ifndef .VARIABLES
> +$(error GNU make 3.80 or newer is required)
> +endif
> with a $(warning) instead, and let it run for a bit.
+1
regards, tom lane
On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote: > On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote: > > Do we have a handle on how many buildfarm members this will break? > > I suppose we don't. One way to find out would be to commit just this > bit > > +# We need the $(eval) function, which is available in GNU make 3.80. > +# That also happens to be the version where the .VARIABLES variable > +# was introduced, so this is a simple check. > +ifndef .VARIABLES > +$(error GNU make 3.80 or newer is required) > +endif > > with a $(warning) instead, and let it run for a bit. So far, two machines have reported an older make version: dawn_bat narwhal both of the mingw type. Andrew, Dave, could you see about upgrading the GNU make installation there? There are a few machines that haven't build in five days or more, but based on their operating system version, it is fairly safe to assume that they have an up-to-date version.
On 11/06/2010 07:35 AM, Peter Eisentraut wrote: > So far, two machines have reported an older make version: > > dawn_bat > narwhal > > both of the mingw type. Andrew, Dave, could you see about upgrading the > GNU make installation there? dawn_bat is done. cheers andrew
On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote: >> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote: >> > Do we have a handle on how many buildfarm members this will break? >> >> I suppose we don't. One way to find out would be to commit just this >> bit >> >> +# We need the $(eval) function, which is available in GNU make 3.80. >> +# That also happens to be the version where the .VARIABLES variable >> +# was introduced, so this is a simple check. >> +ifndef .VARIABLES >> +$(error GNU make 3.80 or newer is required) >> +endif >> >> with a $(warning) instead, and let it run for a bit. > > So far, two machines have reported an older make version: > > dawn_bat > narwhal > > both of the mingw type. Andrew, Dave, could you see about upgrading the > GNU make installation there? Narwhal should be OK now. /D
On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote: > Narwhal should be OK now. The build has issues now, possibly related to the make upgrade.
<br /><br /> On 11/10/2010 10:32 AM, Peter Eisentraut wrote: <blockquote cite="mid:1289403168.13614.0.camel@vanquo.pezone.net"type="cite"><pre wrap="">On tis, 2010-11-09 at 03:54 -0800, Dave Pagewrote: </pre><blockquote type="cite"><pre wrap="">Narwhal should be OK now. </pre></blockquote><pre wrap=""> The build has issues now, possibly related to the make upgrade. </pre></blockquote><br /> Yeah, it's complaining about not finding bison, but configure managed to find bison just fine.Are you sure the right make was installed? It looks suspicious because it's not talking about msys virtual maths likethe old make did. It needs to be <span class="expander" style="width: 27px;"></span><span class="expander" style="width: 27px;"></span>make-3.81-3-msys-1.0.13 <a class="moz-txt-link-rfc2396E" href="http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download"><http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download></a> You'llneed another couple of libraries as well (libiconv and libintl) if they are not already installed. Making this changetook me a while to get right on dawn_bat.<br /><br /> cheers<br /><br /> andrew<br /><br /><br />
On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > Yeah, it's complaining about not finding bison, but configure managed to > find bison just fine. Are you sure the right make was installed? It looks > suspicious because it's not talking about msys virtual maths like the old > make did. It needs to be make-3.81-3-msys-1.0.13 > <http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download> > You'll need another couple of libraries as well (libiconv and libintl) if > they are not already installed. Making this change took me a while to get > right on dawn_bat. I installed the latest make from gnu.org (which I've now uninstalled). The Msys installation on this box is old, and doesn't support the lzma packages used by the latest releases - and from what I can tell, it would take a major upgrade of the installation to get that support. I'm not sure thats a path I want to go down, as I have no idea how much will break if I do that, and I don't exactly have much in the way of spare time to fix it if that happens. I'm currently leaning towards removing the 9.1 build from the machine; on a purely selfish note, I have no interest in mingw/msys builds anymore anyway. However, I'm open to suggestions if anyone knows a relatively safe way to resolve this. /D
On 11/11/2010 06:58 AM, Dave Page wrote: > On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan<andrew@dunslane.net> wrote: >> Yeah, it's complaining about not finding bison, but configure managed to >> find bison just fine. Are you sure the right make was installed? It looks >> suspicious because it's not talking about msys virtual maths like the old >> make did. It needs to be make-3.81-3-msys-1.0.13 >> <http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download> >> You'll need another couple of libraries as well (libiconv and libintl) if >> they are not already installed. Making this change took me a while to get >> right on dawn_bat. > I installed the latest make from gnu.org (which I've now uninstalled). > The Msys installation on this box is old, and doesn't support the lzma > packages used by the latest releases - and from what I can tell, it > would take a major upgrade of the installation to get that support. > I'm not sure thats a path I want to go down, as I have no idea how > much will break if I do that, and I don't exactly have much in the way > of spare time to fix it if that happens. > > I'm currently leaning towards removing the 9.1 build from the machine; > on a purely selfish note, I have no interest in mingw/msys builds > anymore anyway. However, I'm open to suggestions if anyone knows a > relatively safe way to resolve this. No, all you need to unpack those is the basic-bsdtar package. But to save you the pain of all this, I have copied the three objects I installed to get this working on my likewise pretty old Msys to where you can get them. Just grab <http://developer.postgresql.org/~adunstan/msys-make.tgz> As a matter of policy, I do not want to drop support for a FOSS build tool chain on Windows if at all avoidable. cheers andrew
On Thu, Nov 11, 2010 at 1:04 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > No, all you need to unpack those is the basic-bsdtar package. Ahh, OK. That seems to be in the MinGW (compiler) section of the downloads for some reason. > But to save > you the pain of all this, I have copied the three objects I installed to get > this working on my likewise pretty old Msys to where you can get them. Just > grab > <http://developer.postgresql.org/~adunstan/msys-make.tgz> Thanks - installed. > As a matter of policy, I do not want to drop support for a FOSS build tool > chain on Windows if at all avoidable. Nor I, however I only have limited time to dedicate to that goal. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Dave Page wrote: > Thanks - installed. > >> As a matter of policy, I do not want to drop support for a FOSS build tool >> chain on Windows if at all avoidable. > > Nor I, however I only have limited time to dedicate to that goal. One thing to think about is that since PostGIS uses MingW/PGXS on Windows, we use MingW builds in order to generate the Makefiles we need (there is no native MSVC build for Windows). Not being able to do this would cause us great inconvenience :( ATB, Mark. -- Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
On 11/11/2010 11:43 AM, Mark Cave-Ayland wrote: > Dave Page wrote: > >> Thanks - installed. >> >>> As a matter of policy, I do not want to drop support for a FOSS >>> build tool >>> chain on Windows if at all avoidable. >> >> Nor I, however I only have limited time to dedicate to that goal. > > One thing to think about is that since PostGIS uses MingW/PGXS on > Windows, we use MingW builds in order to generate the Makefiles we > need (there is no native MSVC build for Windows). Not being able to do > this would cause us great inconvenience :( > > > Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't its Windows version build with MSVC? cheers andrew
On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > > Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't > its Windows version build with MSVC? Yes - it's a PITA as we have to have a dummy build of the server in mingw/msys to compile PostGIS and Slony. We're probably going to be looking at that in the not-to-distant future as we want 64bit builds of both and will be using VC++. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Dave Page wrote: > On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't >> its Windows version build with MSVC? > > Yes - it's a PITA as we have to have a dummy build of the server in > mingw/msys to compile PostGIS and Slony. We're probably going to be > looking at that in the not-to-distant future as we want 64bit builds > of both and will be using VC++. Just for the record, a lot of work was done in the 1.4 release series to make MSVC builds possible, and indeed several people have reported success: http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html The two main outstanding issues as I see it are: 1) The GTK-based GUI for shp2pgsql (although if someone wanted to sponsor work to convert to wxWidgets to bring us in line with pgAdmin, that would be strongly considered). 2) Maintenance of the MSVC build system. So far we have had some complaints about not using MSVC, but then no-one has stepped up to maintain the build system for it. Forcing all existing developers to suddenly start maintaining the Windows build is a total non-starter. My hope is that one day CMake will enable us to come up with a universal solution, but we're some way from that yet. ATB, Mark. -- Mark Cave-Ayland - Senior Technical Architect PostgreSQL - PostGIS Sirius Corporation plc - control through freedom http://www.siriusit.co.uk t: +44 870 608 0063 Sirius Labs: http://www.siriusit.co.uk/labs
On Thu, Nov 11, 2010 at 5:19 PM, Mark Cave-Ayland <mark.cave-ayland@siriusit.co.uk> wrote: > Dave Page wrote: > >> On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> >> wrote: >>> >>> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and >>> isn't >>> its Windows version build with MSVC? >> >> Yes - it's a PITA as we have to have a dummy build of the server in >> mingw/msys to compile PostGIS and Slony. We're probably going to be >> looking at that in the not-to-distant future as we want 64bit builds >> of both and will be using VC++. > > Just for the record, a lot of work was done in the 1.4 release series to > make MSVC builds possible, and indeed several people have reported success: > > http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html > http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html Cool - that will help. > The two main outstanding issues as I see it are: > > 1) The GTK-based GUI for shp2pgsql (although if someone wanted to sponsor > work to convert to wxWidgets to bring us in line with pgAdmin, that would be > strongly considered). :-) > 2) Maintenance of the MSVC build system. So far we have had some complaints > about not using MSVC, but then no-one has stepped up to maintain the build > system for it. Forcing all existing developers to suddenly start maintaining > the Windows build is a total non-starter. Unless you're making major architectural changes, it shouldn't take any real effort to add/remove the occasional source file. I'm sure there are folks that could be persuaded to do that occasionally. > My hope is that one day CMake will enable us to come up with a universal > solution, but we're some way from that yet. We used CMake for a couple of projects, but ended up abandoning it for new stuff. It just didn't work as nicely as we wanted. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 11/11/2010 03:19 PM, Dave Page wrote: > >> My hope is that one day CMake will enable us to come up with a universal >> solution, but we're some way from that yet. > We used CMake for a couple of projects, but ended up abandoning it for > new stuff. It just didn't work as nicely as we wanted. Yes, it's been discussed before here too and didn't really go anywhere :-( cheers andrew