[Code] Building old toolchains on a new Linux (disabling Fortify)

We used to build our embedded Linux platform on an aged Debian Linux distribution (a 3-year-old vintage). Reently, I moved the build/release servers and my development workstation to Kubuntu 8.10. Shiny new, and still Linux. What could possibly go wrong?

Well, all the old toolchain buildy stuff stopped working.

The toolchain (which was build with the then-aged crosstool) exploded messily during program linkage, with aborts due to heap corruption or double-freeing. I've been scratching my head over this problem for a while.

The telling symptom in the crash backtraces was the function __fortify_fail just before the abort. The kubuntu glibc has been build with Fortify support (the "best" Fortify homepage I can find is a random geocities one; there must be a better one). Fortify is a memory access checker. Very useful to make sure your code is safe to run.

However, the old toolchain we rely on (specifically, the ar in binutils 2.16.1) seems to be badly behaved. It generates good code, but tramples on its own memory whilst it does it. So Fortify kills the process before an executable sees the light of day.

There are three solutions:

• Fix the toolchain
• Port the build system to a newer toolchain
• Switch off fortify checking

Sadly, I don't have time to dig into debugging an archaic binutils right now; and I'm happy from experience that the code it's generating is good. I've been working on migrating our build to a new set of tools and a newer Linux kernel, but if you've ever played with toolchains and build systems you'll know that it's easier said than done. In fact, it's a nightmare, and I just need to get the existing stuff building again right now.

So option 3, it is then. But without any adequate documentation, it's hard to work out out to disable fortify.

The trick is to define a preprocessor variable _FORTIFY_SOURCE=0. Then all glibc memory checking is disabled and the toolchain can continue to not work as well as it did in the past. A handy trick that's almost obvious when you know where to look for it.

Now, how you get that through crosstool to the binutils build stage is left as an excercise for the reader. Suffice to say that our old crosstool is now incredibly bodged and patched up to just work in 2009.