Thread: [patch] build issues on Win32
I've run across a couple of stumbling blocks when building on Win32 (specifically, XP + MinGW): 1. PostgreSQL's private versions of inet_aton etc. can conflict with similar functions in other libraries (in my case, PostgreSQL's inet_aton conflicts with libavformat's). 2. On Win32, PostgreSQL uses the SSPI authentication interface. This interface is implemented in secur32.dll, which needs to be added to the linker command line. This is apparently only an issue when building a static libpq. 3. src/template/win32 sets LDFLAGS unconditionally, overriding anything the user may have specified in the environment or on the command line (such as -L/path/to/my/lib/dir). The attached patch addresses these issues by: 1. #defining the function names as appropriate 2. adding -lsecur32 to LIBS in src/Makefile.global.in when PORTNAME is win32. 3. adding ${LDFLAGS} at the front of the LDFLAGS redefinition in src/template/win32. The patch was developed and tested on 8.3.9, because that's what my customer uses. I have verified that it applies cleanly (albeit with offsets) to 8.4.2. BTW, IWBNI there were a quick and easy way to build and install only libpq. I use this sequence of commands (after configure): $ make -C src/port all $ make -C src/backend utils/fmgroids.h $ make -C src/backend ../../src/include/utils/fmgroids.h $ make -C src/include all install $ make -C src/interfaces/libpq all install $ make -C src/bin/pg_config all install DES -- Dag-Erling Smørgrav - des@des.no --- src/include/port.h.orig 2009-11-14 16:39:41.000000000 +0100 +++ src/include/port.h 2010-03-10 13:17:27.000000000 +0100 @@ -337,6 +337,7 @@ * When necessary, these routines are provided by files in src/port/. */ #ifndef HAVE_CRYPT +#define crypt pq_crypt extern char *crypt(const char *key, const char *setting); #endif @@ -351,44 +352,60 @@ #endif #ifndef HAVE_GETOPT +#define getopt pq_getopt extern int getopt(int nargc, char *const * nargv, const char *ostr); #endif #ifndef HAVE_ISINF +#define isinf pq_isinf extern int isinf(double x); #endif #ifndef HAVE_RINT +#define rint pq_rint extern double rint(double x); #endif #ifndef HAVE_INET_ATON #include <netinet/in.h> #include <arpa/inet.h> +#define inet_aton pq_inet_aton extern int inet_aton(const char *cp, struct in_addr * addr); #endif #ifndef HAVE_STRDUP +#define strdup pq_strdup extern char *strdup(const char *str); #endif +#ifndef HAVE_STRLCAT +#define strlcat pq_strlcat +#endif + #if !HAVE_DECL_STRLCAT extern size_t strlcat(char *dst, const char *src, size_t siz); #endif +#ifndef HAVE_STRLCPY +#define strlcpy pq_strlcpy +#endif + #if !HAVE_DECL_STRLCPY extern size_t strlcpy(char *dst, const char *src, size_t siz); #endif #if !defined(HAVE_RANDOM) && !defined(__BORLANDC__) +#define random pq_random extern long random(void); #endif #ifndef HAVE_UNSETENV +#define unsetenv pq_unsetenv extern void unsetenv(const char *name); #endif #ifndef HAVE_SRANDOM +#define srandom pq_srandom extern void srandom(unsigned int seed); #endif --- src/Makefile.global.in.orig 2007-11-13 01:13:19.000000000 +0100 +++ src/Makefile.global.in 2010-03-10 13:42:04.000000000 +0100 @@ -435,9 +435,10 @@ endif # to make ws2_32.lib the last library, and always link with shfolder, -# so SHGetFolderName isn't picked up from shell32.dll +# so SHGetFolderName isn't picked up from shell32.dll. Also link +# with secur32 for SSPI. ifeq ($(PORTNAME),win32) -LIBS += -lws2_32 -lshfolder +LIBS += -lws2_32 -lshfolder -lsecur32 endif # Not really standard libc functions, used by the backend. --- src/template/win32.orig 2004-11-08 06:23:26.000000000 +0100 +++ src/template/win32 2010-03-10 13:40:59.000000000 +0100 @@ -1,4 +1,4 @@ # This is required to link pg_dump because it finds pg_toupper() in # libpq and pgport -LDFLAGS="-Wl,--allow-multiple-definition" +LDFLAGS="${LDFLAGS} -Wl,--allow-multiple-definition"
Dag-Erling Smørgrav <des@des.no> writes: > I've run across a couple of stumbling blocks when building on Win32 > (specifically, XP + MinGW): > 1. PostgreSQL's private versions of inet_aton etc. can conflict with > similar functions in other libraries (in my case, PostgreSQL's > inet_aton conflicts with libavformat's). So what? We don't link with those libraries. The proposed #defines seem like a completely bad idea, especially since as-presented they would affect every platform not only yours. We don't need the maintenance/debugging headaches of routines that don't have the names they appear to have. > ifeq ($(PORTNAME),win32) > -LIBS += -lws2_32 -lshfolder > +LIBS += -lws2_32 -lshfolder -lsecur32 > endif Surely this bit would need to be conditional on whether libsecur32 is available? > -LDFLAGS="-Wl,--allow-multiple-definition" > +LDFLAGS="${LDFLAGS} -Wl,--allow-multiple-definition" That bit seems sane. regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> writes: > "Dag-Erling Smørgrav" <des@des.no> writes: > > 1. PostgreSQL's private versions of inet_aton etc. can conflict with > > similar functions in other libraries (in my case, PostgreSQL's > > inet_aton conflicts with libavformat's). > So what? We don't link with those libraries. Your users might need to link with both. I'm working on an application that generates animations (specifically, animated weather forecasts) based on data retrieved from a PostgreSQL database. > The proposed #defines seem like a completely bad idea, especially > since as-presented they would affect every platform not only yours. Yes. That's the idea. This is a common idiom - the canonical way, if you will, of solving this problem, which is not exclusive to PostgreSQL. There are even cases in which you have no other choice, e.g. when the function you want to use is available but does not work properly or does not implement a particular feature that you need. > We don't need the maintenance/debugging headaches of routines that > don't have the names they appear to have. Honestly, I don't see the problem. > > ifeq ($(PORTNAME),win32) > > -LIBS += -lws2_32 -lshfolder > > +LIBS += -lws2_32 -lshfolder -lsecur32 > > endif > Surely this bit would need to be conditional on whether libsecur32 > is available? It's just as much part of Win32 as ws2_32 (winsock). DES -- Dag-Erling Smørgrav - des@des.no
2010/3/10 Dag-Erling Smørgrav <des@des.no>: > Tom Lane <tgl@sss.pgh.pa.us> writes: >> "Dag-Erling Smørgrav" <des@des.no> writes: >> > 1. PostgreSQL's private versions of inet_aton etc. can conflict with >> > similar functions in other libraries (in my case, PostgreSQL's >> > inet_aton conflicts with libavformat's). >> So what? We don't link with those libraries. > > Your users might need to link with both. I'm working on an application > that generates animations (specifically, animated weather forecasts) > based on data retrieved from a PostgreSQL database. This shows up only with static links of libpq, correct? Or are you building a backend function? >> The proposed #defines seem like a completely bad idea, especially >> since as-presented they would affect every platform not only yours. > > Yes. That's the idea. This is a common idiom - the canonical way, if > you will, of solving this problem, which is not exclusive to PostgreSQL. > There are even cases in which you have no other choice, e.g. when the > function you want to use is available but does not work properly or does > not implement a particular feature that you need. > >> We don't need the maintenance/debugging headaches of routines that >> don't have the names they appear to have. > > Honestly, I don't see the problem. > >> > ifeq ($(PORTNAME),win32) >> > -LIBS += -lws2_32 -lshfolder >> > +LIBS += -lws2_32 -lshfolder -lsecur32 >> > endif >> Surely this bit would need to be conditional on whether libsecur32 >> is available? > > It's just as much part of Win32 as ws2_32 (winsock). Yeah, it's a standard win32 library so that's not a problem. But the fix seems wrong. If you are using a static libpq, the library should be added whenever you link that library into your client application. Not for every single EXE and DLL that postgres produces. The makefiles already add it to the places in PostgreSQL where it's needed - namely postgres.exe and libpq.dll. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Dag-Erling Smørgrav <des@des.no> writes: > Tom Lane <tgl@sss.pgh.pa.us> writes: >> "Dag-Erling Smørgrav" <des@des.no> writes: >>> 1. PostgreSQL's private versions of inet_aton etc. can conflict with >>> similar functions in other libraries (in my case, PostgreSQL's >>> inet_aton conflicts with libavformat's). >> So what? We don't link with those libraries. > Your users might need to link with both. We don't support linking the backend into other applications. If you're complaining about libpq, the right thing for that is to use a platform that can suppress non-exported symbols from a shared library. Maybe what we need to do is teach the mingw build path how to respect the exports list for libpq? regards, tom lane
Magnus Hagander <magnus@hagander.net> writes: > Dag-Erling Smørgrav <des@des.no> writes: > > Your users might need to link with both. I'm working on an > > application that generates animations (specifically, animated > > weather forecasts) based on data retrieved from a PostgreSQL > > database. > This shows up only with static links of libpq, correct? Yes. > But the fix seems wrong. If you are using a static libpq, the library > should be added whenever you link that library into your client > application. Not for every single EXE and DLL that postgres produces. Without this patch, pg_ctl fails to build... and I don't see the logic of including ws2_32 in LIBS but not secur32. remember that LIBS directly affects what pg_config reports, so if you don't add secur32 to LIBS, consumers (applications) don't know that they need it. DES -- Dag-Erling Smørgrav - des@des.no
Tom Lane <tgl@sss.pgh.pa.us> writes: > We don't support linking the backend into other applications. libpq uses this as well. > If you're complaining about libpq, the right thing for that is to use > a platform that can suppress non-exported symbols from a shared > library. Maybe what we need to do is teach the mingw build path how > to respect the exports list for libpq? If that works, I'm all for it. I have no idea how to do it, though. DES -- Dag-Erling Smørgrav - des@des.no
Dag-Erling Smørgrav <des@des.no> writes: > Magnus Hagander <magnus@hagander.net> writes: >> But the fix seems wrong. If you are using a static libpq, the library >> should be added whenever you link that library into your client >> application. Not for every single EXE and DLL that postgres produces. > Without this patch, pg_ctl fails to build... It builds for everybody else (and we do have multiple mingw machines in the buildfarm, so it's not like this doesn't get tested). I think there is some other factor involved here, and you need to identify what that is. regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> writes: > "Dag-Erling Smørgrav" <des@des.no> writes: > > Without this patch, pg_ctl fails to build... > It builds for everybody else (and we do have multiple mingw machines in > the buildfarm, so it's not like this doesn't get tested). I think there > is some other factor involved here, and you need to identify what that > is. --disable-shared, as previously mentioned. DES -- Dag-Erling Smørgrav - des@des.no
Dag-Erling Smørgrav <des@des.no> writes: > Tom Lane <tgl@sss.pgh.pa.us> writes: >> It builds for everybody else (and we do have multiple mingw machines in >> the buildfarm, so it's not like this doesn't get tested). I think there >> is some other factor involved here, and you need to identify what that >> is. > --disable-shared, as previously mentioned. Oh. Well, we don't really support that, and there is a proposal on the table to remove it altogether from the configure script. I don't think we're going to contort our source code in order to make a marginal improvement in the ability to coexist with random other code that is also polluting the link-time namespace. regards, tom lane
On Wed, Mar 10, 2010 at 01:28:55PM -0500, Tom Lane wrote: > Dag-Erling Smørgrav <des@des.no> writes: > > Tom Lane <tgl@sss.pgh.pa.us> writes: > >> It builds for everybody else (and we do have multiple mingw > >> machines in the buildfarm, so it's not like this doesn't get > >> tested). I think there is some other factor involved here, and > >> you need to identify what that is. > > > --disable-shared, as previously mentioned. > > Oh. Well, we don't really support that, and there is a proposal on > the table to remove it altogether from the configure script. I > don't think we're going to contort our source code in order to make > a marginal improvement in the ability to coexist with random other > code that is also polluting the link-time namespace. +1 for de-supporting this option. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
2010/3/10 David Fetter <david@fetter.org>: >> > --disable-shared, as previously mentioned. >> >> Oh. Well, we don't really support that, and there is a proposal on >> the table to remove it altogether from the configure script. I >> don't think we're going to contort our source code in order to make >> a marginal improvement in the ability to coexist with random other >> code that is also polluting the link-time namespace. > > +1 for de-supporting this option. I would be sad about this. It seems likely there are platforms where it's important. But I'm not really about to spend the effort to fix it up myself and I agree it wouldn't be worth hacking the source to get it to work. I'm a bit puzzled why the symbol conflicts occur only with static linking though -- it seems like static linking would give more opportunity to isolate symbols than dynamic linking, not less. Perhaps our static linking rules are broken? -- greg
Greg Stark <gsstark@mit.edu> writes: > I would be sad about this. It seems likely there are platforms where > it's important. But I'm not really about to spend the effort to fix it > up myself and I agree it wouldn't be worth hacking the source to get > it to work. I'm a bit puzzled why the symbol conflicts occur only with > static linking though -- it seems like static linking would give more > opportunity to isolate symbols than dynamic linking, not less. Perhaps > our static linking rules are broken? A dynamic library is a single entity with a certain degree of isolation. You can hide symbols so they are only visible within that library. A static library is basically just a tarball of individual relocatable objects. The GNU toolchain (and probably others too) allows you to combine several relocatable objects together into one and hide some of the symbols, as if the entire thing had been a single C file with some of the functions declared as static. However, when you do that, you lose an important advantage of static libraries: the ability to link only what you need. DES -- Dag-Erling Smørgrav - des@des.no
2010/3/11 Dag-Erling Smørgrav <des@des.no>: > Greg Stark <gsstark@mit.edu> writes: >> I would be sad about this. It seems likely there are platforms where >> it's important. But I'm not really about to spend the effort to fix it >> up myself and I agree it wouldn't be worth hacking the source to get >> it to work. I'm a bit puzzled why the symbol conflicts occur only with >> static linking though -- it seems like static linking would give more >> opportunity to isolate symbols than dynamic linking, not less. Perhaps >> our static linking rules are broken? > > A dynamic library is a single entity with a certain degree of isolation. > You can hide symbols so they are only visible within that library. In particular, libpq only exports a fixed subset of symbols on any platform that supports that (which includes Windows). AFAIK, there is no way to make that restriction on static libraries, at least not on Windows. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes: > In particular, libpq only exports a fixed subset of symbols on any > platform that supports that (which includes Windows). > AFAIK, there is no way to make that restriction on static libraries, > at least not on Windows. Right, and not anyplace else either except by pre-linking the contained .o files, which eliminates one of the claimed advantages of a .a library. So one of the reasons not to support a static version of libpq is precisely this one: it means that every non-static symbol inside libpq is subject to conflicts against calling applications and other libraries that they choose to link with. I don't particularly care to deal with that, and I especially don't choose to accept the position that any such conflicts are *our* problem to fix. regards, tom lane
Greg Stark <gsstark@mit.edu> writes: > 2010/3/10 David Fetter <david@fetter.org>: >>>> --disable-shared, as previously mentioned. >> >> +1 for de-supporting this option. > I would be sad about this. It seems likely there are platforms where > it's important. Any such platform would already be contending with plpgsql not working, encoding conversion not working, etc etc. It's barely conceivable that a client-only installation would be useful. But given that nobody has actually proposed supporting such a platform in the past ten years, I don't think one's likely to come out of the woodwork now. AFAICT the only case where anyone tries to do this is they have a personal preference to avoid shared libraries, for generally-pretty-darn-dubious reasons. Red Hat has developed a pretty strict policy against even shipping static libraries, because it's such a PITA to deal with updates. Let me give you a fresh-in-mind example: there is an open security bug against libpng (which I package for Red Hat in my copious spare time). I was distressed to find that fixing the bug left firefox still failing against a test web page. Investigation disclosed that the reason for this is that firefox is using a private static copy of libpng. That was a stupid decision on their part, as it means they're going to have to be involved in fixing this issue, not to mention past and future issues. Now libpq doesn't often have critical security bugs filed against it, but it certainly has bugs. Do you really want to have to remember to rebuild every piece of dependent software when you update it? The OP's case apparently involves multiple independent libraries that he wants to link statically, which makes the problem multiple times worse. So my position is that static linking has enough negatives that you need a lot more than a hypothetical use-case to justify it. regards, tom lane
2010/3/11 Tom Lane <tgl@sss.pgh.pa.us>: > Now libpq doesn't often have critical security bugs filed against it, > but it certainly has bugs. Do you really want to have to remember to > rebuild every piece of dependent software when you update it? I absolutely agree that linking statically is a terrible idea for distributions. Debian has a similar prohibition against linking statically against any other package. But it does ship static libraries because commercial binary-only packages often require them. They have to be packaged to work on multiple OS distributions including some which don't provide especially smooth dependency systems. You don't want to get calls from customers running your software against random library versions you've never tested. The two (separate) goals which are useful are 1) Provide a library others can link against to produce a binary which has no run-time dependency on your library. In this case the only library they might want to link against would be libpq. The encoding libraries etc aren't things they're going to link agains. And 2) build binaries which have no dependencies on system libraries so someone can ship them and expect them to run on any system regardless of the run-time environment. I agree that these are both over-used but they are sometimes the least bad option. On the other hand, the third goal "avoid using the shared library facilities" is pointless, I see no reason to avoid building and loading the encoding code or the contrib modules. They're using the same technology as shared libraries but they're not really shared libraries in the sense of being shipped separately from the binaries using them. -- greg
Tom Lane wrote: > Now libpq doesn't often have critical security bugs filed against it, > but it certainly has bugs. Do you really want to have to remember to > rebuild every piece of dependent software when you update it? The OP's > case apparently involves multiple independent libraries that he wants to > link statically, which makes the problem multiple times worse. > > So my position is that static linking has enough negatives that you > need a lot more than a hypothetical use-case to justify it. > > +1. cheers andrew
Greg Stark <gsstark@mit.edu> writes: > The two (separate) goals which are useful are 1) Provide a library > others can link against to produce a binary which has no run-time > dependency on your library. In this case the only library they might > want to link against would be libpq. The encoding libraries etc aren't > things they're going to link agains. And 2) build binaries which have > no dependencies on system libraries so someone can ship them and > expect them to run on any system regardless of the run-time > environment. > I agree that these are both over-used but they are sometimes the least > bad option. On the other hand, the third goal "avoid using the shared > library facilities" is pointless, I see no reason to avoid building > and loading the encoding code or the contrib modules. They're using > the same technology as shared libraries but they're not really shared > libraries in the sense of being shipped separately from the binaries > using them. True. That still makes the current --disable-shared configure option useless, but it doesn't go as far as suggesting that we ought to implement --disable-static, which would be the logical conclusion from my position. I don't actually want to do --disable-static, but my feeling about it is "if it breaks you get to keep both pieces". I'm not interested in providing workarounds to let a static libpq be used in combination with any-random-other-static-library. That's up to the user of the thing to deal with. BTW, I'm not sure I buy the argument that commercial software requires static linking. Red Hat would be as interested in that market as anybody, and as I said, they don't think it's necessary to ship static libraries (with a *very* short list of exceptions). regards, tom lane
On Mar 11, 2010, at 1:06 PM, Tom Lane wrote: > > > BTW, I'm not sure I buy the argument that commercial software requires > static linking. Red Hat would be as interested in that market as > anybody, and as I said, they don't think it's necessary to ship static > libraries (with a *very* short list of exceptions). Trying to support a binary distributed[1] application on a dozen or more different distributions is fairly painful (I'm building on four Debians, eight Ubuntus, several RHELs, three Solarises and a few others right now), and the biggest problem is the different distributions don't package all the libraries you need, and when they do they include a wide range of versions, often woefully obsolete. It's a serious pain. The "woefully obsolete" bit also means that you have to code to the lowest common denominator - you cannot use any feature more recent than the obsolete library that the oldest version of RHEL you support has, nor any feature that's been deprecated in the latest bleeding edge Ubuntu release (There are some APIs where an old feature has been replaced by a new feature, and the old one deprecated and removed over a timeframe short enough that conservative distributions only have the old one and more bleeding edge distributions only have the new one. libpq is stable enough and backwards compatible enough that it's never been an issue there). Building applications mostly statically linked avoids the vast majority of those problems. It's not the only way to do it, and I don't think it's ever the best way to do it as it can lead to a bunch of other problems but it's often the easiest fix for getting cross-platform support working right now, and I can understand why people want to be able to do it. Cheers, Steve [1] Binary distributed is not the same as commercial, though a lot of commercial code is distributed as binaries so it's one place where the problem comes up. Commercial software is also more limited in it's ability to just blame the OS vendor and ignore the issue.
On Thu, Mar 11, 2010 at 01:34:34PM -0800, Steve Atkins wrote: > On Mar 11, 2010, at 1:06 PM, Tom Lane wrote: > > > > BTW, I'm not sure I buy the argument that commercial software > > requires static linking. Red Hat would be as interested in that > > market as anybody, and as I said, they don't think it's necessary > > to ship static libraries (with a *very* short list of exceptions). > > Trying to support a binary distributed[1] application on a dozen or > more different distributions is fairly painful (I'm building on four > Debians, eight Ubuntus, several RHELs, three Solarises and a few > others right now), and the biggest problem is the different > distributions don't package all the libraries you need, and when > they do they include a wide range of versions, often woefully > obsolete. I've found it easiest to ship all of PostgreSQL rather than deal with this kind of stuff. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Tom Lane <tgl@sss.pgh.pa.us> writes: > Any such platform would already be contending with plpgsql not working, > encoding conversion not working, etc etc. It's barely conceivable that > a client-only installation would be useful. Uh, I don't understand why it's so hard to conceive that someone might need the client library to access a database running on one machine from an application running on another. > AFAICT the only case where anyone tries to do this is they have a > personal preference to avoid shared libraries, for generally-pretty- > darn-dubious reasons. Well, statically linked binaries are much easier to distribute, for one; and there are platforms where shared libraries simply do not exist, or where the entire system runs in a single namespace. Netware is still alive and kicking. DES -- Dag-Erling Smørgrav - des@des.no
2010/3/12 Dag-Erling Smørgrav <des@des.no>: > Tom Lane <tgl@sss.pgh.pa.us> writes: >> Any such platform would already be contending with plpgsql not working, >> encoding conversion not working, etc etc. It's barely conceivable that >> a client-only installation would be useful. > > Uh, I don't understand why it's so hard to conceive that someone might > need the client library to access a database running on one machine from > an application running on another. Well, in this specific case, we're talking Windows, right? Why not just distribute a libpq.dll along with the application, and you're done? It's not that people don't realize it may be useful. Just that it's not useful enough to enough people to put lots of workarounds in th ecode to make it work. >> AFAICT the only case where anyone tries to do this is they have a >> personal preference to avoid shared libraries, for generally-pretty- >> darn-dubious reasons. > > Well, statically linked binaries are much easier to distribute, for one; > and there are platforms where shared libraries simply do not exist, or > where the entire system runs in a single namespace. Netware is still > alive and kicking. Do we even support netware on liboq? :-) -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Wed, Mar 10, 2010 at 10:51:23AM -0500, Tom Lane wrote: > Dag-Erling Smørgrav <des@des.no> writes: > > -LDFLAGS="-Wl,--allow-multiple-definition" > > +LDFLAGS="${LDFLAGS} -Wl,--allow-multiple-definition" > > That bit seems sane. I've committed that part. Thanks. -- Noah Misch EnterpriseDB http://www.enterprisedb.com