Context

Last week i have been working on getting a particularly complex beast to be cross-compilable under Void Linux's xbps-src: GObject Introspection.

While work was done on xbps-src cross compilation of CPython3 modules, 2 problems were found.

The first problem, which will be discussed and fixed in this topic, is that modules cross-compiled used the ABI extension tag in their .so files. Example: .cpython-36m-x86_64-linux-gnu.so for x86_64.

That value is defined by the PEP 3149 and conveniently is available as the EXT_SUFFIX variable in the sysconfig module.

Examples for x86_64 and aarch64:

$ python3
>>> import sysconfig
>>> sysconfig.get_config_var('EXT_SUFFIX')
'.cpython-36m-x86_64-linux-gnu.so'

$ python3
>>> import sysconfig
>>> sysconfig.get_config_var('EXT_SUFFIX')
'.cpython-36m-aarch64-linux-gnu.so'

Problem

The problem occurs when cross compiling, the python3 used belongs to the build system and not of the target system, consequently the EXT_SUFFIX used is the build system one, not of the target one as it should be.

As an example, cross-compiling the python-yaml package with xbps-src will produce the binaries with the wrong name:

$ xbps-src -a aarch64 pkg python-yaml
$ env XBPS_TARGET_ARCH=aarch64 xls python3-yaml | grep .cpython
/usr/lib/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so

See what's wrong ? A package compiled for aarch64 (ARM 64bits) has the prefix of x86_64. That's not right!

Let's push this further and actually try to load another package that uses CPython on aarch64, python-bluez!

$ python3
>>> import bluetooth
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/terran/usr/src/void-packages/masterdir/usr/aarch64-linux-gnu/usr/lib/python3.6/site-packages/bluetooth/__init__.py", line 45, in <module>
    from bluetooth.bluez import *
  File "/home/terran/usr/src/void-packages/masterdir/usr/aarch64-linux-gnu/usr/lib/python3.6/site-packages/bluetooth/bluez.py", line 10, in <module>
    import bluetooth._bluetooth as _bt
ModuleNotFoundError: No module named 'bluetooth._bluetooth'

That is borked!, but do not despair, lets remove the EXT_SUFFIX.

$ mv -v usr/lib/python3.6/site-packages/bluetooth/_bluetooth{.cpython-36m-x86_64-linux-gnu,}.so

And let's try again!

$ python3
>>> import bluetooth
>>>

Success! So our solution is simply to just remove the EXT_SUFFIX from all modules.

Solving

For this xbps-src has a very nifty mechanism, hooks.

Hooks are files with the .sh extension that provide a hook() function, depending on where the file is located it will be run during a specific stage of the package build process.

As an example: hooks in common/hooks/post-install will run after the do_install() function is ran, these hooks are available for all major stages of building a package, before (pre-<stage>) during (do-<stage>) and after (post-<stage>).

So let's just write a hook that does the renaming for us!

# This hook executes the following tasks:
#	- renames cpython binding files to not include the arch-specific extension suffix

hook() {
    if [ ! -d ${PKGDESTDIR}/${py3_sitelib} ]; then
        return 0
    fi

    find "${PKGDESTDIR}/${py3_sitelib}" -type f -executable -iname '*.cpython*.so' \
        | while read -r file; do
        filename="${file##*/}"
        modulename="${filename%%.*}"
        msg_warn "${pkgver}: renamed '${filename}' to '${modulename}.so'.\n"
        mv ${file} ${file%/*}/${modulename}.so
    done
}

Heh, looks good enough to me. Improvements can be made with a pull request to void-packages.

Let's try building our package again!

$ xbps-src -a aarch64 pkg python-yaml
=> WARNING: python3-yaml-3.13_2: renamed '_yaml.cpython-36m-x86_64-linux-gnu.so' to '_yaml.so'.
$ env XBPS_TARGET_ARCH=aarch64 xls python3-yaml | grep '.so$'
/usr/lib/python3.6/site-packages/_yaml.so

Nice! Our work here is done. Meet me up in pt. 2 for a similar story around another problem.