Introduction

On 2019-01-20, starting with this commit, the groundwork was laid for a new mechanism inside xbps-src that will prove to be immensely helpful in tackling new challenges like, cross compiling rust and GObject Introspection.

Overview

On its own

build_helper is a mechanism to source files location in common/build-helper that end with .sh. A template just needs to include it by setting the build_helper variable, like so:

build_helper="foo bar"

This will make the files foo.sh and bar.sh be sourced, variables exported are going on the global environment and can affect the behaviour of applications, as we will see in the examples.

vs. build_style

build_helper can be thought of as a less powerful version of the commonly used build_style with the advantage that multiple build_helper can be used on the same template.

While build_style can define functions for the stages of packaging, like do_build(), do_install(), do_fech(), etc. build_helper can only define environment variables and template specific variables like hostmakedepends.

In a sense, build_helpers act as an capsule for complex and/or repetitive behaviour that is not needed all the time, costly operations that are specific to one domain like cross compiling rust

Creation

Solving a problem

A long time ago we started making rust cross compilable, we split rust-std into its own package, which can be installed on makedepends and be built with the cargo and rustc from hostmakedepends.

At that time most of the rust code that was being cross compiled as present solely in the form of crates we could fetch with cargo.

Simple enough, we would just create the cargo build_style, define some fancy functions to do all the work with cargo and getting it to compile by exporting some variables.

It was done and everything was happily thereafter, all packages that were pure-rust and used cargo were now cross compilable.

But then we ran into mixed code, people liked rust and people were slowly rewriting their programs in rust, Firefox as the sponsors of the language were also the front-runners, but other projects like librsvg, a very important library that is dependended by a lot of packages. And newsboat a small RSS reader i happen to be the Void Linux maintainer of also started rewriting.

With the mixed code we couldn't just plug in the cargo build_style, those packages had their own build systems that demanded their own build_style, librsvg uses gnu-configure , newsboat uses configure and Firefox has their own "thing".

We needed to export lots of variables to affect behaviour, luckily they all used cargo under the hood to compile the rust code so we could make use of cargo's environment configuration variables.

First attempt

The first attempt was with expanding the scope of build_style, instead of sourcing files in common/environment/build-style/<build_style>.sh it would now have its own directory and would source all the files inside.

It could now have different files for functions and for environment variables and the latter could be shared via symlinks, suddenly the cargo build_style could share the environment variables it was using with the gnu-configure one for newsboat and librsvg.

This idea came crashing down when a very inherent flaw was pointed out by another Void dev

I don't think build styles need to be more complex.

The cargo configuration shouldn't depend on the build_style because at the end any buildstyle could need it, librsvg and firefox are examples that use no build_style and gnu-configure.

second attempt

With the first implementation soundly destroyed, a second attempt was drafted.

One important objective of the second attempt was to isolate the complexity, we needed the complexity because we were tackling complex problems but we also wanted to avoid it whenever we were not tackling those complex problems.

The solution came after someone suggested creating a build_style but for specific environments on the #xbps IRC freenode. And thus build_helper was born, with this pull request as implementation.

Example

Getting rusty

This is the rust build_helper, it defines some important variables to us, like the target which we will build it for and other variables that allow us to build -sys crates, those crates are built using the system components provided by Void Linux.

# Define equivalent of TOML config in environment
# [build]
# jobs = $XBPS_MAKEJOBS
export CARGO_BUILD_JOBS="$XBPS_MAKEJOBS"

if [ "$CROSS_BUILD" ]; then
    # Define equivalent of TOML config in environment
    # [target.${RUST_TARGET}]
    # linker = ${CC}
    _XBPS_CROSS_RUST_TARGET_ENV="${XBPS_CROSS_RUST_TARGET^^}"
    _XBPS_CROSS_RUST_TARGET_ENV="${_XBPS_CROSS_RUST_TARGET_ENV//-/_}"
    export CARGO_TARGET_${_XBPS_CROSS_RUST_TARGET_ENV}_LINKER="$CC"
    unset _XBPS_CROSS_RUST_TARGET_ENV
    
    # Define equivalent of TOML config in environment
    # [build]
    # target = ${RUST_TARGET}
    export CARGO_BUILD_TARGET="$RUST_TARGET"
fi

# For cross-compiling rust -sys crates
export PKG_CONFIG_ALLOW_CROSS=1

# libgit2-sys
export LIBGIT2_SYS_USE_PKG_CONFIG=1

# gettext-rs
export GETTEXT_BIN_DIR=/usr/bin
export GETTEXT_LIB_DIR="${XBPS_CROSS_BASE}/usr/lib/gettext"
export GETTEXT_INCLUDE_DIR="${XBPS_CROSS_BASE}/usr/include"

# libssh2-sys
export LIBSSH2_SYS_USE_PKG_CONFIG=1

Before it existed all that complexity was either tied to the cargo build_style or was polluting the global environment.

This is the newsboat template

# Template file for 'newsboat'
pkgname=newsboat
version=r2.14
revision=1
build_style=configure
build_helper="rust"
configure_script="./config.sh"
make_build_target="all doc"
make_install_args="prefix=/usr"
hostmakedepends="asciidoc pkg-config cargo"
makedepends="json-c-devel libcurl-devel libxml2-devel sqlite-devel stfl-devel
 rust-std"
checkdepends="ncurses-base"
short_desc="Newsboat is a fork of Newsbeuter the Mutt of RSS feed readers"
maintainer="maxice8 <thinkabit.ukim@gmail.com>"
license="MIT"
homepage="https://www.newsboat.org"
changelog="https://raw.githubusercontent.com/newsboat/newsboat/master/CHANGELOG.md"
distfiles="https://github.com/newsboat/newsboat/archive/${version}.tar.gz"
checksum=57176071b0c777ddba237f95f684e36bf2ddf271e84ac5c9e3eec8f1afc0b133

CXXFLAGS=" -Wno-error=sign-compare"

if [ "$CROSS_BUILD" ]; then
    LDFLAGS="-L./target/${RUST_TARGET}/release"
fi

pre_configure() {
    if [ "$CROSS_BUILD" ]; then
        cd doc
        g++ -o generate generate.cpp
        g++ -o generate2 generate2.cpp
        g++ -o gen-example-config gen-example-config.cpp
    fi
}

do_check() {
    rm -f test/rss.cpp

    make test
    (cd test && TERM=$TERM TMPDIR=/dev/shm ./test)
}

post_install() {
    vlicense LICENSE
    vsconf doc/example-bookmark-plugin.sh bookmark-plugin.sh
    mv "${DESTDIR}/usr/share/doc/$pkgname/examples/config" "${DESTDIR}/usr/share/examples/${pkgname}"
}

As you can see with the rust build_helper, the only thing we needed was to pass an extra location to LDFLAGS to search find the newsboat static archive. Under the hood all the magic needed to compile rust code to another architecture and to use system components when building -sys crates was dealt with.

Future Introspection

Another use case that is being dealt with is cross compiling with GObject Introspection.

It isn't available as of the time this blog post was written but is instead held up in this pull request.

Cross compiling deserves a post on their own (wink wink), so only the build_helper being introduced to help it, gir, is going to be gleaned over.

Have at it:

#
# gir - build-helper for gobject-introspection
#
# This build-helper is used for packages that make use of
# the GObject introspection middleware layer.
# 

# Check if the 'gir' build_option is set or if there is no
# 'gir' build_option.
if [ "$build_option_gir" ] || [[ $build_options != *"gir"* ]]; then
    # Provide the host tooling, g-ir-scanner, g-ir-compiler and its
    # wrappers.
    hostmakedepends+=" gobject-introspection"
    
    if [ "$CROSS_BUILD" ]; then
        # Required for running binaries produced from g-ir-compiler
        # via g-ir-scanner-qemuwrapper
        hostmakedepends+=" qemu-user-static"
    
        # Required for running the g-ir-scanner-lddwrapper
        hostmakedepends+=" prelink-cross"

        # Provide basic .gir types like GLib, GObject, DBus, Gio, cairo
        # and tooling like g-ir-compiler
        makedepends+=" gobject-introspection"
    fi
fi

As observable, the build_helper will gladly add all the dependencies required for both native and cross compilation, it will even be careful to not add them if we have a build_option called gir it also needs to be enabled.