When writing makefiles goes wrong
Introduction
Recently giflib made a new update, after a long time, from 5.1.4 to 5.1.5, it didn't have any impressive at first, fixing a few bugs then i read this.
Build Fixes ----------- The horrible old autoconf build system has been removed with extreme prejudice. You now build this simply by running "make" from the top-level directory.
I thought to myself, oh nice, it makes stuff simpler, less packages for us to install to build the package means reduced build times and since we just build and install we don't need incremental compiling and all that other nice stuff.
Wrong!
My other side went, it is extremely easy to fuck up Makefiles though, let's hope they did their homework.
They didn't.
Here is the TLDR for those that do not have time or just don't want to take the journey with us.
Updating
Let's start by updating it, seems simple enough at first, we change the build_style from gnu-configure to gnu-makefile, we update version, reset revision to 1 and update the checksum.
diff --git a/srcpkgs/giflib/template b/srcpkgs/giflib/template index d11669ebaf..9369f96b66 100644 --- a/srcpkgs/giflib/template +++ b/srcpkgs/giflib/template @@ -1,15 +1,15 @@ # Template file for 'giflib' pkgname=giflib -version=5.1.4 +version=5.1.5 revision=1 -build_style=gnu-configure +build_style=gnu-makefile hostmakedepends="xmlto" short_desc="Library to handle, display and manipulate GIF images" maintainer="Juan RP <xtraeme@voidlinux.org>" homepage="http://sourceforge.net/projects/giflib/" license="MIT" distfiles="${SOURCEFORGE_SITE}/${pkgname}/${pkgname}-${version}.tar.gz" -checksum=34a7377ba834397db019e8eb122e551a49c98f49df75ec3fcc92b9a794a4f6d1 +checksum=b105046665089149fc16e20a59f54e70ee829b61168a46881f5037469ff4058f post_install() { vlicense COPYING LICENSE
Looks good enough to me, let's try to build it.
Missing CFLAGS
/usr/bin/ld: getarg.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value collect2: error: ld returned 1 exit status
Ah, they forgot to set -fPIC on their CFLAGS when building the shared library.
No problem, let's add it ourselves.
diff --git a/srcpkgs/giflib/template b/srcpkgs/giflib/template index d11669ebaf..62712456f8 100644 --- a/srcpkgs/giflib/template +++ b/srcpkgs/giflib/template @@ -9,7 +9,9 @@ maintainer="Juan RP <xtraeme@voidlinux.org>" homepage="http://sourceforge.net/projects/giflib/" license="MIT" distfiles="${SOURCEFORGE_SITE}/${pkgname}/${pkgname}-${version}.tar.gz" -checksum=34a7377ba834397db019e8eb122e551a49c98f49df75ec3fcc92b9a794a4f6d1 +checksum=b105046665089149fc16e20a59f54e70ee829b61168a46881f5037469ff4058f + +CFLAGS="-fPIC" post_install() { vlicense COPYING LICENSE
And let's build it again
Missing usr/include
mv: cannot stat '/destdir//giflib-5.1.5/usr/include': No such file or directory => ERROR: giflib-devel-5.1.5_1: pkg_install: 'mv ${_destdir}/$files ${_pkgdestdir}/${_targetdir}' exited with 1 => ERROR: in _vmove() at common/environment/setup/install.sh:231 => ERROR: in _noglob_helper() at common/environment/setup/install.sh:12 => ERROR: in pkg_install() at srcpkgs/giflib-devel/template:24
What ?!, It didn't install the /usr/include headers ?!
masterdir/destdir/giflib-devel-5.1.5/ └── usr 1 directory, 0 files
Huh ?!, Let's take a look at their makefile.
Missing DESTDIR
# Installation/uninstallation install: all install-bin install-man install-bin: $(INSTALLABLE) $(INSTALL) -d "$(PREFIX)/bin" $(INSTALL) $^ "$(PREFIX)/bin" install-man: $(INSTALL) -d "$(PREFIX)/share/man/man1" $(INSTALL) -m 644 doc/*.1 "$(PREFIX)/share/man/man1" uninstall: uninstall-man uninstall-bin uninstall-bin: cd "$(PREFIX)/bin"; rm $(INSTALLABLE) uninstall-man: cd $(target)/share/man/man1/ && rm -f $(shell cd doc >/dev/null ; echo *.1)<Paste>
They don't even try to install the headers! And they don't respect the DESTDIR variable so they install against the host system not against the fakeroot we have!
Let's just check the giflib-tools package which should contain the binaries.
masterdir/destdir/giflib-5.1.5/ └── usr └── share └── licenses └── giflib └── LICENSE 4 directories, 1 file
There is nothing, except the license, the license we install manually via vlicense not
their Makefile!
Let's correct this, now
diff --git a/Makefile b/Makefile index d197193..ea62148 100644 --- Makefile +++ Makefile @@ -83,11 +83,11 @@ make check: all install: all install-bin install-man install-bin: $(INSTALLABLE) - $(INSTALL) -d "$(PREFIX)/bin" - $(INSTALL) $^ "$(PREFIX)/bin" + $(INSTALL) -d "$(DESTDIR)$(PREFIX)/bin" + $(INSTALL) $^ "$(DESTDIR)$(PREFIX)/bin" install-man: - $(INSTALL) -d "$(PREFIX)/share/man/man1" - $(INSTALL) -m 644 doc/*.1 "$(PREFIX)/share/man/man1" + $(INSTALL) -d "$(DESTDIR)$(PREFIX)/share/man/man1" + $(INSTALL) -m 644 doc/*.1 "$(DESTDIR)$(PREFIX)/share/man/man1" uninstall: uninstall-man uninstall-bin uninstall-bin: cd "$(PREFIX)/bin"; rm $(INSTALLABLE)
And also install the header
@@ -14,6 +14,11 @@ checksum=b105046665089149fc16e20a59f54e70ee829b61168a46881f5037469ff4058f CFLAGS="-fPIC" post_install() { + # Their new makefile is broken and doesn't install lots of essential + # components + vmkdir usr/include + vcopy gif_lib.h usr/include + vlicense COPYING LICENSE }
OK, another time to build.
Missing static archive
mv: cannot stat '/destdir//giflib-5.1.5/usr/lib/*.a': No such file or directory => ERROR: giflib-devel-5.1.5_1: pkg_install: 'mv ${_destdir}/$files ${_pkgdestdir}/${_targetdir}' exited with 1 => ERROR: in _vmove() at common/environment/setup/install.sh:231 => ERROR: in _noglob_helper() at common/environment/setup/install.sh:12 => ERROR: in pkg_install() at srcpkgs/giflib-devel/template:28
OK, now the static archive is not available, you know what ? We don't need them anyway for this specific package, let's just remove it.
@@ -22,7 +25,6 @@ giflib-devel_package() { short_desc+=" - development files" pkg_install() { vmove usr/include - vmove usr/lib/*.a vmove usr/lib/*.so } }
And another time.
Missing shared library
mv: cannot stat '/destdir//giflib-5.1.5/usr/lib/*.so': No such file or directory => ERROR: giflib-devel-5.1.5_1: pkg_install: 'mv ${_destdir}/$files ${_pkgdestdir}/${_targetdir}' exited with 1 => ERROR: in _vmove() at common/environment/setup/install.sh:231 => ERROR: in _noglob_helper() at common/environment/setup/install.sh:12 => ERROR: in pkg_install() at srcpkgs/giflib-devel/template:28
OK, now the solink the -devel package is missing, for those that don't know the solink is a symlink to the library file that has no soversion in it.
As an example:
- libfoo.so.1.0.0 (library file)
- libfoo.so.1 (soname symlink)
- libfoo.so (solink)
Packages when linked, look for the soname symlink, when an ABI breaking change is made the soname is bumped, in this case from libfoo.so.1 to libfoo.so.2 and all packages that depend on libfoo need to be rebuilt to search for the correct library.
The solink is there for the development package and is used when linking.
Back to giflib, which readers at #xbps, Void Linux's development
channel know started annoying me.
Well, let's fix it ourselves, just install it as libgif.so.7.1.0 (value taken from their Makefile) and the proper symlinks, should be easy enough.
@@ -17,6 +17,11 @@ post_install() { vmkdir usr/include vcopy gif_lib.h usr/include + vmkdir usr/lib + vinstall giflib.so 0755 usr/lib libgif.so.7.1.0 + ln -rs ${DESTDIR}/usr/lib/libgif.so.7.1.0 ${DESTDIR}/usr/lib/libgif.so.7 + ln -rs ${DESTDIR}/usr/lib/libgif.so.7.1.0 ${DESTDIR}/usr/lib/libgif.so + vlicense COPYING LICENSE }
OK, building one more time:
=> Registering new packages to /host/binpkgs/giflib2 index: added `giflib-5.1.5_1' (x86_64). index: added `giflib-devel-5.1.5_1' (x86_64). index: added `giflib-tools-5.1.5_1' (x86_64). index: 3 packages registered.
Huzzah! It compiled, luckily i use a package that uses giflib so it is good to test it.
Missing SONAME
# xbps-src --repository=hostdir/binpkgs/giflib -u giflib imlib2-1.5.1_1: broken, unresolvable shlib `libgif.so.7' Transaction aborted due to unresolved shlibs.
Heh ? It can't find the libgif.so.7 ?!, i'm pretty sure we installed it, let's check it again just to be sure
$ xbps-query --repository=hostdir/binpkgs/giflib -f giflib /usr/lib/libgif.so.7.1.0 /usr/share/licenses/giflib/LICENSE /usr/lib/libgif.so.7 -> /usr/lib/libgif.so.7.1.0
It is there! Why can't it find it ?!, Let's compare the 2 packages with xbps-query
the one from the repos and the one that we built. Thanks Bash for <() super helpful
diff <(xbps-query -RS giflib) <(xbps-query --repository=hostdir/binpkgs/giflib -S giflib)
2,4c2,4 < build-date: 2016-04-02 23:14 UTC < filename-sha256: e0c1c6bd50e3a618db99d5c583133c5aae65b905c6c11e44b54980404879b735 < filename-size: 16KB --- > build-date: 2019-02-13 05:55 -02 > filename-sha256: ea59be29e8801dc02b6deb305440903c072307dc634a0616ce779201ced655ca > filename-size: 19KB 6c6 < installed_size: 39KB --- > installed_size: 47KB 8,12c8,10 < maintainer: Juan RP <xtraeme@voidlinux.eu> < pkgver: giflib-5.1.4_1 < repository: https://alpha.de.repo.voidlinux.org/current < shlib-provides: < libgif.so.7 --- > maintainer: Juan RP <xtraeme@voidlinux.org> > pkgver: giflib-5.1.5_1 > repository: /home/terran/usr/src/void-packages/hostdir/binpkgs/giflib2 16d13 < source-revisions: giflib:b54c327
Looking at it, build-date, filename-sha256, filename-size, installed_size, maintainer, pkgver and repository all are ok to change, but one thing is missing.
There is no shlib-provides! No wonder it fails, the soname wasn't registered.
Missing -Wl,-soname
Fine then, let's find out how shlib-provides is made, to do that we need to look
at another hook from xbps-src hooks/pre-pkg/06-shlib-provides.sh
Let's zoom in at the section that matters.
# real pkg find ${_destdir} -type f -name "*.so*" | while read f; do _fname="${f##*/}" case "$(file -bi "$f")" in application/x-sharedlib*|application/x-pie-executable*) # shared library _soname=$(${OBJDUMP} -p "$f"|grep SONAME|awk '{print $2}') # Register all versioned sonames, and # unversioned sonames only when in libdir. if [[ ${_soname} =~ ${_versioned_pattern} ]] || [[ ${_soname} =~ ${_pattern} && ( -e ${_destdir}/usr/lib/${_fname} || -e ${_destdir}/usr/lib32/${_fname} ) ]]; then echo "${_soname}" >> ${_tmpfile} echo " SONAME ${_soname} from ${f##${_destdir}}" fi ;; esac done
OK let's check a few things, let's use file -bi to see if libgif.so.7.1.0 matches the case. Using xbps-src install giflib to install it to masterdir/destdir for our testing.
$ file -bi masterdir/destdir/giflib-5.1.5/usr/lib/libgif.so.7.1.0 application/x-pie-executable; charset=binary
OK, it matches now the next thing, it uses objdump -p and greps for the SONAME field
$ objdump -p masterdir/destdir/giflib-5.1.5/usr/lib/libgif.so.7.1.0 | grep SONAME
Nothing ?!, just to be sure we will check if that field really matters against another random library on the system that is present, libnotmuch.
$ objdump -p /usr/lib/libnotmuch.so.5.2.0 | grep SONAME SONAME libnotmuch.so.5
And the shlib-provides ?
$ xbps-query -RS libnotmuch -p shlib-provides libnotmuch.so.5
Alright, we are missing the SONAME field, but how do we get it ? According to this we need to pass -Wl,-soname,libgif.so.7 to the linker flags of the library.
Before we do that let's check how giflib creates the shared library, back to their makefile.
giflib.so: $(OBJECTS) $(HEADERS) $(CC) $(CFLAGS) -shared $(OFLAGS) -o giflib.so $(OBJECTS) ln -sf giflib.so giflib.so.$(LIBMAJOR).$(LIBMINOR).$(LIBPOINT) ln -sf giflib.so giflib.so.$(LIBMAJOR)
Fancy, no LDFLAGS but they do present a OFLAGS for us, let's patch it in.
diff --git a/Makefile b/Makefile index ea62148..43408a9 100644 --- Makefile +++ Makefile @@ -62,6 +62,8 @@ all: giflib.so giflib.a $(UTILS) $(UTILS):: giflib.a +override OFLAGS += -Wl,-soname,libgif.so.$(LIBMAJOR) + giflib.so: $(OBJECTS) $(HEADERS) $(CC) $(CFLAGS) -shared $(OFLAGS) -o giflib.so $(OBJECTS) ln -sf giflib.so giflib.so.$(LIBMAJOR).$(LIBMINOR).$(LIBPOINT)
Building it again, let's keep a close eye for that specific hook and check our shlib-provides and check for the soname.
=> giflib-5.1.5_1: running pre-pkg hook: 06-shlib-provides ... SONAME libgif.so.7 from /usr/lib/libgif.so.7.1.0
$ xbps-src --repository=hostdir/binpkgs/giflib -S giflib -p shlib-provides libgif.so.7
$ objdump -p masterdir/destdir/giflib-5.1.5/usr/lib/libgif.so.7.1.0 | grep SONAME SONAME libgif.so.7
Checking the result
Seems to be all good. Now to install it again.
# xbps-install --repository=hostdir/binpkgs/giflib -u giflib Do you want to continue? [Y/n] 1 package will be updated: giflib (5.1.4_1 -> 5.1.5_1) Size required on disk: 8696B Space available on disk: 183GB Do you want to continue? [Y/n] y [*] Downloading binary packages [*] Verifying package integrity giflib-5.1.5_1: verifying SHA256 hash... [*] Running transaction tasks giflib-5.1.4_1: updating to 5.1.5_1 ... giflib-5.1.5_1: unpacking ... giflib-5.1.5_1: removed obsolete entry: ./usr/lib/libgif.so.7.0.0 [1A[K [*] Configuring unpacked packages giflib-5.1.5_1: configuring ... giflib-5.1.5_1: updated successfully. 0 downloaded, 0 installed, 1 updated, 0 configured, 0 removed.
Mission complete.
Final recap
Let's recap and make a TLDR for people that do not have the time to take this journey with us.
What giflib upstream did:
- Changed their build system from gnu-configure to gnu-makefile
What went wrong:
- Didn't respect DESTDIR when installing stuff
- Makes it harder for distros, since they install to a fakeroot and pack it up
- Didn't set -fPIC when compiling the shared library
- Caused compilation failure when compiling the shared library
- Didn't install development headers
- Required for developing against giflib
- Didn't install static archive
- Required for building static pacakges that use giflib
- Didn't install shared library
- Required for packages that link against giflib like emacs
- Didn't set -Wl,-soname when building the shared library
- Prevents the system linker from working it all out and breaks xbps
Basically everything that could have gone wrong went wrong.
Promoting Meson
This is where i shill hard promote my favorite build system,
meson where it is made harder to mess up
up like it was done with giflib, unlike Makefiles where messing up
is the natural order and people are most of the time do wrong a lot
before they get it right once.
Upstreaming
maxice8 are you going to upstream it ?
No, they are behind sourceforge, if they were on Github/Gitlab or even a self hosted instance of gitlab or another cooperative git interface i would do it eagerly.
So yeah feel free to upstream all that stuff if you feel like it.