Fixing cross CPython3 modules in Void Linux, pt. 2
Context
After fixing the problem discussed in part 1. Another bug was hit.
$ python3 >>> import bluetooth ImportError: /usr/lib/python3.6/site-packages/bluetooth/_bluetooth.so: undefined symbol: PyString_FromStringAndSize
After googling that specific error i was made aware that PyString_FromStringAndSize
is not available on python3.
But why then ? Why did it get included in the python3 build ? Let's take a
look at the source.
$ xbps-src -I extract python-bluez $ cd masterdir/builddir/pybluez-0.22 $ rg --no-ignore PyString_FromStringAndSize bluez/btmodule.c 664: PyObject *buf = PyString_FromStringAndSize((char *)NULL, buflen); 1073: buf = PyString_FromStringAndSize((char *) 0, len); 1123: buf = PyString_FromStringAndSize((char *) 0, len); 1949: return PyString_FromStringAndSize(rparam, req.rlen); 2124: return PyString_FromStringAndSize(param, len); \ 2153: return PyString_FromStringAndSize(param, len); \ port3/port3.h 5: #define PyString_FromStringAndSize PyBytes_FromStringAndSize osx/_osxbt.c 190: buf = PyString_FromStringAndSize((char*)0, datalen); 456: rawrecord = PyString_FromStringAndSize( qs->lpBlob->pBlobData, msbt/_msbt.c 292: buf = PyString_FromStringAndSize((char*)0, datalen); 646: rawrecord = PyString_FromStringAndSize( qs->lpBlob->pBlobData,
Interesting results, specially that port/port3.h, let's take a closer look
$ cat port3/port3.h
#if PY_MAJOR_VERSION >= 3 #define PyInt_FromLong PyLong_FromLong #define PyString_FromString PyUnicode_FromString #define PyString_FromStringAndSize PyBytes_FromStringAndSize #define _PyString_Resize _PyBytes_Resize #define PyString_AS_STRING PyBytes_AS_STRING #define PyInt_AsLong PyLong_AsLong #define PyString_AsString PyBytes_AsString #define PyInt_Check PyLong_Check #define PyInt_AS_LONG PyLong_AS_LONG #define BYTES_FORMAT_CHR "y#" #else #ifndef Py_TYPE #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) #endif #define BYTES_FORMAT_CHR "s#" #endif
So for some reason, PY_MAJOR_VERSION is not set correctly. After some googling it was revealed that PY_MAJOR_VERSION is defined by python headers, more specifically the patchlevel.h header.
$ rg --no-ignore PY_MAJOR_VERSION /usr/include/python3.6m /usr/include/python3.6m/patchlevel.h 19:#define PY_MAJOR_VERSION 3 31:#define PY_VERSION_HEX ((PY_MAJOR_VERSION << 24) | \
Problem
Well, let's take a look at our include directives, maybe the correct one isn't being included. Let's see the build logs.
aarch64-linux-gnu-gcc -pthread -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python2.7 -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python3.6m -I/usr/aarch64-linux-gnu/usr/include -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/usr/aarch64-linux-gnu/usr/lib -L/usr/aarch64-linux-gnu/lib/python2.7 -L/usr/aarch64-linux-gnu/usr/lib -L/usr/aarch64-linux-gnu/lib/python3.6 -L/usr/aarch64-linux-gnu/usr/lib -DNDEBUG -g -fwrapv -O3 -Wall -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python2.7 -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python3.6m -I/usr/aarch64-linux-gnu/usr/include -fPIC -I./port3 -I/usr/include/python3.6m -c bluez/btsdp.c -o build-3.6/temp.linux-x86_64-3.6/bluez/btsdp.o
This has too much content, lets remove everything but the include directives, remove duplicates and break one per line to better examine them.
aarch64-linux-gnu-gcc -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python2.7 -I/usr/aarch64-linux-gnu/include/python3.6m -I./port3
Much clearer, and it is now visible why the wrong header was included.
The python2 headers are being included before the python3 ones,
PY_MAJOR_VERSION is always being set to 2.
To find out why includes meant for python2 are getting inserted into the python3
build i turned to the build style.
For those uninitiated, build styles are snippets in shell that define functions and environment variables for dealing with specific build systems, like cmake and meson and specific languages like go and python.
In the case of python-bluez the build style being used is python-module,
which builds the package for python2 and python3 in separate build directories.
Let's then take a peek into common/build-style/python-module.
do_build() { : ${python_versions:="2.7 $py3_ver"} local pyver= pysufx= for pyver in $python_versions; do if [ -n "$CROSS_BUILD" ]; then PYPREFIX="$XBPS_CROSS_BASE" if [ "$pyver" != "2.7" ]; then pysufx=m fi CFLAGS+=" -I${XBPS_CROSS_BASE}/include/python${pyver}${pysufx} -I${XBPS_CROSS_BASE}/usr/include" LDFLAGS+=" -L${XBPS_CROSS_BASE}/lib/python${pyver} -L${XBPS_CROSS_BASE}/usr/lib" CC="${XBPS_CROSS_TRIPLET}-gcc -pthread $CFLAGS $LDFLAGS" LDSHARED="${CC} -shared $LDFLAGS" env CC="$CC" LDSHARED="$LDSHARED" \ PYPREFIX="$PYPREFIX" CFLAGS="$CFLAGS" \ LDFLAGS="$LDFLAGS" python${pyver} setup.py \ build --build-base=build-${pyver} ${make_build_args} else python${pyver} setup.py build --build-base=build-${pyver} ${make_build_args} fi done }
Noticed it ?, CFLAGS and LDFLAGS are being appended. But they are global variables
and will not reset between each loop. That means that after building for python2 it
will have a polluted environment for building for python3!
Solution
This should be an easy fix... Just store the CFLAGS and LDFLAGS at the start of the
function as another variable and reset between each loop!
diff --git a/common/build-style/python-module.sh b/common/build-style/python-module.sh index 57ec8c73831..d3c93de38ca 100644 --- a/common/build-style/python-module.sh +++ b/common/build-style/python-module.sh @@ -4,10 +4,13 @@ do_build() { : ${python_versions:="2.7 $py3_ver"} - local pyver= pysufx= + local pyver= pysufx= tmp_cflags="$CFLAGS" tmp_ldflags="$LDFLAGS" for pyver in $python_versions; do if [ -n "$CROSS_BUILD" ]; then + CFLAGS="$tmp_cflags" + LDFLAGS="$tmp_ldflags" + PYPREFIX="$XBPS_CROSS_BASE" if [ "$pyver" != "2.7" ]; then pysufx=m
And done, let's build it again and take a look at the build logs and perform the same cleanup done as the first time for readability.
aarch64-linux-gnu-gcc -I/usr/aarch64-linux-gnu/usr/include -I/usr/aarch64-linux-gnu/include/python3.6m -I./port3
And let's run the module again and see what happens
$ python3 >>> import bluetooth >>>
Success! Another bug squashed! With this the path to having python-gobject available to cross arches almost complete! Onwards to making gobject-introspection cross!