Misidentification of Python shared library - Mailing list pgsql-hackers

While chasing down last night's failure on buildfarm member longfin,
I came across an interesting factoid.  If you just do
./configure --enable-sharedmakemake install

in unmodified Python sources, what you will get is an install tree in
which libpython.so (or local equivalent such as .dylib) is installed
in /usr/local/lib, while libpython.a is installed in a directory named
like /usr/local/lib/python3.5/config-3.5m.  I've verified this behavior
on both Linux and macOS.

The trouble is that we are picking up /usr/local/lib/python3.5/config-3.5m
as the python_libdir.  This means that we think we are using a python
shared library (because we see that Python was configured to build one)
but what we are actually linking with is a static library.  This happens
to work, because the code in the static library was built with -fpic,
but it results in an undesirably bloated plpython.so.

Or at least it did work until commit d51924be8.  With that, plpython.so
and hstore_plpython.so are each bloated with their very own copy of the
Python code.  And they fail to interoperate at all, which is why longfin
crashed.  I didn't investigate very hard, but it looks to me like Python
assumes that symbols like Py_None will have unique addresses, which they
don't in this scenario.

There's a comment in config/python.m4 that correctly describes what
Python is exposing:

# Note: selecting libpython from python_configdir works in all Python
# releases, but it generally finds a non-shared library, which means
# that we are binding the python interpreter right into libplpython.so.
# In Python 2.3 and up there should be a shared library available in
# the main library location.

but the code below this is not coming to the right conclusions, at least
not in Python 3.5.  What I'm seeing on macOS is that we get
"libpython3.5m.dylib" for python_ldlibrary, which seems right, but
python_so comes out as ".cpython-35m-darwin.so", whereas the code looks
like it's expecting that to be ".dylib".  The lack of match causes
the if-test just below that to make the wrong choice.  So some sort of
fix is needed here.

Also, I propose that we'd be better off to not rely on Py_ENABLE_SHARED
at all, but just to run the code in configure.in that insists on
finding something in $python_libdir that is named like a shared library,
because installation trees like this mean that Py_ENABLE_SHARED is
totally untrustworthy as a guide to what is actually in any particular
library directory.  Moreover, this stanza:
   if test "$PORTNAME" = darwin; then     # macOS does supply a .dylib even though Py_ENABLE_SHARED does     # not get
set. The file detection logic below doesn't succeed     # on older macOS versions, so make it explicit.
python_enable_shared=1

is completely misguided because it presumes we are using the
Apple-supplied python library and not a hand-installed one.  I think
the similar exception for Windows is probably bogus for the same reason.

In short: I propose replacing all of this logic with "if there's something
in $python_libdir that has the right name to be a python shared library,
use that, else try the same in $python_configdir, else fail".  Thoughts?
        regards, tom lane