Sometimes something simple, like adding a Python dependency, turns out to be unexpectedly difficult.

We recently tried to include python-saml in our Heroku build. Sounds easy enough; it depends on libxmlsec, we just install that with apt-get and everything should work fine… except Heroku doesn’t let us use apt-get directly.

No problems. As it so happens there’s a Heroku buildpack for precisely this situation: heroku-buildpack-xmlsec. Combined with sequential buildpacks this seems like the perfect solution. Sadly not to be, as we found with this error:

ImportError: /app/.heroku/python/lib/python3.5/site-packages/xmlsec/constants.cpython-35m-x86_64-linux-gnu.so: undefined symbol: xmlSecXPath2Ns

This is a fairly daunting looking error, and can potentially mean a number of things have gone wrong, the most likely of which is that python-saml was unable to locate the xmlsec library. Logging on to Heroku and checking the library links with

ldd -D /app/.heroku/python/lib/python3.5/site-packages/xmlsec/constants.cpython-35m-x86_64-linux-gnu.so

confirmed that libxmlsec.so was not being found.

Motivated by the possibility of a quick solution, a different, more general, and hopefully better supported buildpack was tried: heroku-buildpack-apt. Sadly it did not provide any progress.

Investigating python-saml’s source code showed it uses pkg-config to locate the paths for libxmlsec. I noticed it won’t fail loudly if it can’t find the libraries, which explains why we didn’t know about the missing libraries until actually trying to run something. It seems the issue is now most likely to do with setting pkg-config values.

Next I opened up heroku-buildpack-apt to see how it goes about building packages. Turns out it downloads the deb package, extract it somewhere and then builds it in a non-standard location. This has the effect of invalidating any prefixes in the pkg-config files. So, how do we fix this? The easiest solution was to modify the buildpack to rewrite *.pc files with the appropriate prefixes:

topic "Rewrite package-config files"
find $BUILD_DIR/.apt -type f -ipath '*/pkgconfig/*.pc' | xargs -n 1 sed -i -e 's!^prefix=\(.*\)$!prefix='"$BUILD_DIR"'/.apt\1!g'

However when trying again that same error is back. I can see all appropriate variables are exported from heroku-buildpack-apt, so perhaps they aren’t finding their way to heroku-buildpack-python?

The Heroku documentation says that to pass variables to subsequent buildpacks we use a file called export. Checking heroku-buildpack-apt shows it is indeed exporting correctly. This leaves the probable location of the error in…

heroku-buildpack-python seems to stomp on the environment variables handed to it from previous buildpacks. After some light tweaking everything worked as expected (see below for links to the forked buildpacks).

Lessons:

1. Fail early, fail loudly.
2. Deb packages don’t like non-standard locations.

Code:

• heroku-buildpack-apt
• heroku-buildpack-python