python header puzzle

python header puzzle

[Marco Atzeri]

Hi All,

trying to build postgresql 12.X with python3 interface, I hit an
interesting issue

all headers
/usr/include/python{3.6-3.8}/pyport.h

are including

#                       if defined(__CYGWIN__)
#                               define PyMODINIT_FUNC 
__declspec(dllexport) PyObject*
#                       else /* __CYGWIN__ */
#                               define PyMODINIT_FUNC PyObject*
#                       endif /* __CYGWIN__ */

and the postgresql python 3 interface has in the code and the header
src/pl/plpython/plpy_plpymodule.{h,c}

the following lines:

PyMODINIT_FUNC PyInit_plpy(void);

PyMODINIT_FUNC PyInit_plpy(void)...

Unfortunately the shared lib produced is crippled and does not
export correctly the symbols.

Changing the lines in

extern PyObject* PyInit_plpy(void);

PyObject* PyInit_plpy(void)..

solves the problem and pass all tests.

At first glance the Python headers seem the wrong one; the postgresql
code has no special  CYGWIN handling in that point, just the note

  * Must have external linkage, because PyMODINIT_FUNC does dllexport on
  * Windows-like platforms.

Comments ?
Any one seeing similar issue with Python3 code ?

For what I see the usage of PyMODINIT_FUNC is very limited
only python-cffi and python-lxml seem to use it in their headers


Regards

Marco

Re: python header puzzle

[Lemures Lemniscati]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Fri, 1 Jan 2021 19:33:03 +0100, Marco Atzeri via Cygwin
> Hi All,
> 
> trying to build postgresql 12.X with python3 interface, I hit an
> interesting issue
> 
> all headers
> /usr/include/python{3.6-3.8}/pyport.h
> 
> are including
> 
> #                       if defined(__CYGWIN__)
> #                               define PyMODINIT_FUNC __declspec(dllexport) PyObject*
> #                       else /* __CYGWIN__ */
> #                               define PyMODINIT_FUNC PyObject*
> #                       endif /* __CYGWIN__ */
> 
> and the postgresql python 3 interface has in the code and the header
> src/pl/plpython/plpy_plpymodule.{h,c}
> 
> the following lines:
> 
> PyMODINIT_FUNC PyInit_plpy(void);
> 
> PyMODINIT_FUNC PyInit_plpy(void)...
> 
> Unfortunately the shared lib produced is crippled and does not
> export correctly the symbols.
> 
> Changing the lines in
> 
> extern PyObject* PyInit_plpy(void);
> 
> PyObject* PyInit_plpy(void)..
> 
> solves the problem and pass all tests.
> 
> At first glance the Python headers seem the wrong one; the postgresql
> code has no special  CYGWIN handling in that point, just the note
> 
>   * Must have external linkage, because PyMODINIT_FUNC does dllexport on
>   * Windows-like platforms.
> 
> Comments ?
> Any one seeing similar issue with Python3 code ?
> 
> For what I see the usage of PyMODINIT_FUNC is very limited
> only python-cffi and python-lxml seem to use it in their headers


How about like this, though it should need more refinements.

A cygport file is attached, or placed at
https://github.com/cygwin-lem/cygwin-pkg/tree/w_postgresql-12.5-1/postgresql

Regards,

Lem


[-- Attachment #2: postgresql.cygport --]
[-- Type: application/octet-stream, Size: 8664 bytes --]

NAME="postgresql"
VERSION=12.5
RELEASE=1

CATEGORY="Database"
SUMMARY="object-relational SQL database management system"
DESCRIPTION="object-relational SQL database management system"
HOMEPAGE="https://www.postgresql.org/"
SRC_URI="https://ftp.postgresql.org/pub/source/v${PV}/${PN}-${PV}.tar.bz2"
SRC_DIR="${PN}-${PV}"

BUILD_REQUIRES="\
  libcrypt-devel\
  libintl-devel\
  libreadline-devel\
  libssl-devel\
  openldap\
  openldap-devel\
  perl_base\
  python38-devel\
  zlib-devel\
"


PATCH_URI="
postgresql-cygwin.patch
cygwin-soversion.diff"

DIFF_EXCLUDES="*.sql"
RESTRICT="debuginfo"
ls
MAKEOPTS="${MAKEOPTS} -j1"
PACKAGE_BUGREPORT="cygwin@cygwin.com"
CYGCONF_ARGS="--enable-nls --with-openssl --with-perl --with-python --with-ldap --with-system-tzdata=/usr/share/zoneinfo"

PYTHON="python3.8"

DEPS_PATH=/usr/sbin


PKG_NAMES="${PN} ${PN}-devel ${PN}-client ${PN}-contrib ${PN}-doc \
  ${PN}-plperl ${PN}-plpython \
  libpq5 libpq-devel libpgtypes3 libecpg6 libecpg-compat3 libecpg-devel"

postgresql_SUMMARY="PostgreSQL Data Base Management System"
postgresql_REQUIRES="postgresql-client crypt openssl" 
postgresql_devel_SUMMARY="development files for PostgreSQL server-side programming"
postgresql_devel_CATEGORY="Devel Database"
postgresql_client_SUMMARY="front-end programs for PostgreSQL"
postgresql_client_REQUIRES="crypt openssl"
postgresql_contrib_SUMMARY="additional facilities for PostgreSQL"
postgresql_doc_SUMMARY="documentation for PostgreSQL"
postgresql_plperl_SUMMARY="PL/Perl procedural language for PostgreSQL"
postgresql_plperl_REQUIRES="postgresql perl"
postgresql_plpython_SUMMARY="PL/Python procedural language for PostgreSQL"
postgresql_plpython_REQUIRES="postgresql python"
libpq5_SUMMARY="PostgreSQL C client library"
libpq5_REQUIRES="crypt openssl"
libpq5_CATEGORY="Database Libs"
libpq_devel_SUMMARY="header files for libpq5 (PostgreSQL C client library)"
libpq_devel_REQUIRES="openssl-devel"
libpq_devel_CATEGORY="Database Libs Devel"
libpgtypes3_SUMMARY="shared library pgtypes for PostgreSQL"
libpqtypes3_CATEGORY="Database Libs"
libecpg6_SUMMARY="run-time library for ECPG (Embedded PostgreSQL for C)"
libecpg6_CATEGORY="Database Libs"
libecpg_compat3_SUMMARY="run-time library for ECPG compact(Embedded PostgreSQL for C)"
libecpg_compat3_CATEGORY="Database Libs"
libecpg_devel_SUMMARY="development files for ECPG (Embedded PostgreSQL for C)"
libecpg_devel_CATEGORY="Database Libs Devel"

postgresql_CONTENTS=" \
--exclude=usr/share/postgresql/contrib \
--exclude=usr/share/postgresql/psqlrc.sample \
etc \
usr/sbin/initdb.exe \
usr/sbin/pg_controldata.exe \
usr/sbin/pg_ctl.exe \
usr/bin/pg_waldump.exe \
usr/sbin/postgres.exe \
usr/sbin/postmaster \
usr/lib/postgresql/*_and_*.dll \
usr/lib/postgresql/dict_snowball.dll \
usr/lib/postgresql/euc2004_sjis2004.dll \
usr/lib/postgresql/libpqwalreceiver.dll \
usr/lib/postgresql/plpgsql.dll \
usr/share/locale/*/LC_MESSAGES/initdb-12.mo \
usr/share/locale/*/LC_MESSAGES/pg_controldata-12.mo \
usr/share/locale/*/LC_MESSAGES/pg_ctl-12.mo \
usr/share/locale/*/LC_MESSAGES/pg_waldump-12.mo \
usr/share/locale/*/LC_MESSAGES/postgres-12.mo \
usr/share/man/man1/initdb.1.gz \
usr/share/man/man1/pg_controldata.1.gz \
usr/share/man/man1/pg_ctl.1.gz \
usr/share/man/man1/pg_waldump.1.gz \
usr/share/man/man1/postgres.1.gz \
usr/share/man/man1/postmaster.1.gz \
usr/share/doc/Cygwin/${PN}.README \
usr/share/postgresql"

postgresql_devel_CONTENTS=" \
usr/include/postgresql/server \
usr/lib/postgresql/pgxs \
usr/lib/libpostgres.a \
usr/share/man/man3"

# TODO: usr/lib/pkgconfig/

postgresql_client_CONTENTS=" \
--exclude=usr/share/locale/*/LC_MESSAGES/initdb-12.mo \
--exclude=usr/share/locale/*/LC_MESSAGES/pg_controldata-12.mo \
--exclude=usr/share/locale/*/LC_MESSAGES/pg_ctl-12.mo \
--exclude=usr/share/locale/*/LC_MESSAGES/pg_waldump-12.mo \
--exclude=usr/share/locale/*/LC_MESSAGES/postgres-12.mo \
--exclude=usr/share/locale/*/LC_MESSAGES/pg_config-12.mo \
usr/bin/clusterdb.exe \
usr/bin/createdb.exe \
usr/bin/dropdb.exe \
usr/bin/pg_archivecleanup.exe \
usr/bin/pg_basebackup.exe \
usr/bin/pg_checksums.exe \
usr/bin/pg_dump.exe \
usr/bin/pg_dumpall.exe \
usr/bin/pg_isready.exe \
usr/bin/pg_receivewal.exe \
usr/bin/pg_recvlogical.exe \
usr/bin/pg_resetwal.exe \
usr/bin/pg_rewind.exe \
usr/bin/pg_test_fsync.exe \
usr/bin/pg_test_timing.exe \
usr/bin/pg_upgrade.exe \
usr/bin/pgbench.exe \
usr/bin/psql.exe \
usr/bin/reindexdb.exe \
usr/sbin/createuser.exe \
usr/sbin/dropuser.exe \
usr/sbin/pg_restore.exe \
usr/sbin/vacuumdb.exe \
usr/share/man/man1/clusterdb.1.gz \
usr/share/man/man1/createdb.1.gz \
usr/share/man/man1/createuser.1.gz \
usr/share/man/man1/dropdb.1.gz \
usr/share/man/man1/dropuser.1.gz \
usr/share/man/man1/pg_checksums.1.gz \
usr/share/man/man1/pg_dump.1.gz \
usr/share/man/man1/pg_dumpall.1.gz \
usr/share/man/man1/pg_basebackup.1.gz \
usr/share/man/man1/pg_isready.1.gz \
usr/share/man/man1/pg_receivewal.1.gz \
usr/share/man/man1/pg_recvlogical.1.gz \
usr/share/man/man1/pg_resetwal.1.gz \
usr/share/man/man1/pg_restore.1.gz \
usr/share/man/man1/pg_rewind.1.gz \
usr/share/man/man1/psql.1.gz \
usr/share/man/man1/reindexdb.1.gz \
usr/share/man/man1/vacuumdb.1.gz \
usr/share/man/man1/vacuumlo.1.gz \
usr/share/man/man7 \
usr/share/postgresql/psqlrc.sample \
usr/share/locale/*/LC_MESSAGES/"

postgresql_contrib_CONTENTS=" \
--exclude=usr/lib/postgresql/*_and_*.dll \
--exclude=usr/lib/postgresql/dict_snowball.dll \
--exclude=usr/lib/postgresql/euc2004_sjis2004.dll \
--exclude=usr/lib/postgresql/libpqwalreceiver.dll \
--exclude=usr/lib/postgresql/pgxs \
--exclude=usr/lib/postgresql/pl*.dll \
usr/lib/postgresql \
usr/share/doc/${PN}/contrib \
usr/share/man/man1/oid2name.1.gz \
usr/share/man/man1/pg_archivecleanup.1.gz \
usr/share/man/man1/pg_standby.1.gz \
usr/share/man/man1/pg_test_fsync.1.gz \
usr/share/man/man1/pg_test_timing.1.gz \
usr/share/man/man1/pg_upgrade.1.gz \
usr/share/man/man1/pgbench.1.gz
"

postgresql_doc_CONTENTS=" \
--exclude=usr/share/doc/${PN}/contrib \
usr/share/doc/${PN}"

postgresql_plperl_CONTENTS="usr/lib/postgresql/plperl.dll"

postgresql_plpython_CONTENTS="usr/lib/postgresql/plpython3.dll"

libpq5_CONTENTS="usr/bin/cygpq-5.dll"

libpq_devel_CONTENTS=" \
usr/bin/pg_config.exe \
usr/include/libpq-fe.h \
usr/include/libpq-events.h \
usr/include/libpq/libpq-fs.h \
usr/include/pg_config*.h \
usr/include/pgtypes.h \
usr/include/postgres_ext.h \
usr/include/postgresql/internal \
usr/lib/libpgport.a \
usr/lib/libpgport_shlib.a \
usr/lib/libpq.a \
usr/lib/libpgcommon.a \
usr/lib/libpgcommon_shlib.a \
usr/lib/libpgfeutils.a \
usr/lib/pkgconfig/libpq.pc \
usr/share/man/man1/pg_config.1.gz \
usr/share/locale/*/LC_MESSAGES/pg_config-12.mo"

libpgtypes3_CONTENTS="usr/bin/cygpgtypes-3.dll"

libecpg6_CONTENTS=" \
usr/bin/ecpg.exe \
usr/bin/cygecpg-6.dll \
usr/share/man/man1/ecpg.1.gz"

libecpg_compat3_CONTENTS="usr/bin/cygecpg_compat-3.dll"

libecpg_devel_CONTENTS=" \
usr/include/ecpg*.h \
usr/include/postgresql/informix \
usr/include/pgtypes_*.h \
usr/include/sql3types.h \
usr/include/sqlca.h \
usr/include/sqlda*.h \
usr/lib/libecpg.a \
usr/lib/libecpg_compat.a \
usr/lib/libpgtypes.a \
usr/lib/pkgconfig/libecpg.pc \
usr/lib/pkgconfig/libecpg_compat.pc \
usr/lib/pkgconfig/libpgtypes.pc" 

src_compile() {
  export PYTHON
  export LDFLAGS="-Wl,-no-undefined -Wl,-R/usr/lib"
  cd ${B}
  cygconf
  cygmake
  cd contrib
  cygmake 
}

src_test() {
  cd ${B}
  CYGWIN="ntsec binmode server" \
  PATH="${D}/usr/sbin:${PATH}" \
  make -j1 -i  MAX_CONNECTIONS=10 check  
  cd contrib
  CYGWIN="ntsec binmode server" \
  PATH="${D}/usr/sbin:${PATH}" \
  make -j1 -i  MAX_CONNECTIONS=10 check  
}

src_install() {
  cd ${B}
  # defining MAKE_DLL to install also libpostgres.a
  # if somemone want to link on internals 
  cyginstall MAKE_DLL=true docdir=/usr/share/doc/${PN}/

  mv ${D}/usr/lib/*.dll ${D}/usr/bin/
  dodir /usr/sbin
  for admin in createlang createuser droplang dropuser initdb \
               pg_ctl pg_controldata pg_resetxlog pg_restore postgres \
               postmaster vacuumdb; do
     mv ${D}/usr/bin/$admin.exe ${D}/usr/sbin/;
  done

  dodir /etc/rc.d/init.d
  cp ${C}/postgresql.initd ${D}/etc/rc.d/init.d/postgresql
  chmod +x ${D}/etc/rc.d/init.d/postgresql

  cp -r ${S}/doc/src/html ${D}/usr/share/doc/${PN}

  dodir /usr/share/man/man1
  cp ${S}/doc/src/sgml/man1/*.1  ${D}/usr/share/man/man1
  dodir /usr/share/man/man3
  cp ${S}/doc/src/sgml/man3/*.3  ${D}/usr/share/man/man3
  dodir /usr/share/man/man7
  cp ${S}/doc/src/sgml/man7/*.7  ${D}/usr/share/man/man7

  make -j1 -C contrib install DESTDIR=${D} bindir=/usr/lib/postgresql/bin/ docdir=/usr/share/doc/${PN}/contrib
}

Re: python header puzzle

[Lemures Lemniscati]

On Sat, 02 Jan 2021 07:30:05 +0900, Lemures Lemniscati
> On Fri, 1 Jan 2021 19:33:03 +0100, Marco Atzeri via Cygwin
> > Hi All,
> > 
> > trying to build postgresql 12.X with python3 interface, I hit an
> > interesting issue
> > 
> > all headers
> > /usr/include/python{3.6-3.8}/pyport.h
> > 
> > are including
> > 
> > #                       if defined(__CYGWIN__)
> > #                               define PyMODINIT_FUNC __declspec(dllexport) PyObject*
> > #                       else /* __CYGWIN__ */
> > #                               define PyMODINIT_FUNC PyObject*
> > #                       endif /* __CYGWIN__ */
> > 
> > and the postgresql python 3 interface has in the code and the header
> > src/pl/plpython/plpy_plpymodule.{h,c}
> > 
> > the following lines:
> > 
> > PyMODINIT_FUNC PyInit_plpy(void);
> > 
> > PyMODINIT_FUNC PyInit_plpy(void)...
> > 
> > Unfortunately the shared lib produced is crippled and does not
> > export correctly the symbols.
> > 
> > Changing the lines in
> > 
> > extern PyObject* PyInit_plpy(void);
> > 
> > PyObject* PyInit_plpy(void)..
> > 
> > solves the problem and pass all tests.
> > 
> > At first glance the Python headers seem the wrong one; the postgresql
> > code has no special  CYGWIN handling in that point, just the note
> > 
> >   * Must have external linkage, because PyMODINIT_FUNC does dllexport on
> >   * Windows-like platforms.
> > 
> > Comments ?
> > Any one seeing similar issue with Python3 code ?
> > 
> > For what I see the usage of PyMODINIT_FUNC is very limited
> > only python-cffi and python-lxml seem to use it in their headers
> 
> 
> How about like this, though it should need more refinements.
> 
> A cygport file is attached, or placed at
> https://github.com/cygwin-lem/cygwin-pkg/tree/w_postgresql-12.5-1/postgresql
> 
I'm sorry. I was wrong.

Regards,

Lem

Re: python header puzzle

[Marco Atzeri]

On 02.01.2021 03:13, Lemures Lemniscati via Cygwin wrote:
> On Sat, 02 Jan 2021 07:30:05 +0900, Lemures Lemniscati
>> On Fri, 1 Jan 2021 19:33:03 +0100, Marco Atzeri via Cygwin
>>> Hi All,
>>>
>>> trying to build postgresql 12.X with python3 interface, I hit an
>>> interesting issue
>>>

>>
>> A cygport file is attached, or placed at
>> https://github.com/cygwin-lem/cygwin-pkg/tree/w_postgresql-12.5-1/postgresql
>>
> I'm sorry. I was wrong.
> 
> Regards,
> 
> Lem

Hi Lem,
Maybe next time; just remember to trim replies.

I am currently using python-cffi to see is I should modify
PyMODINIT_FUNC in the python code or in the packages.

There are other additional puzzling cripped DLL cases.

I am using
https://www.dependencywalker.com/
to verify if the symbol list is correcly exported. When crippled there
is usually only one symbol

Regards
Marco