Tuesday 27 May 2008

Unit testing threads is hard

Unsurprising fact that people don't talk about that much #10045: Unit testing threads is hard.

Unit tests are good. Threads are good. At least, these days many people are telling us that they are good. Threads are the inevitable future of your programming career, if you still want to eat lunch.

But unit tests and threads are not a good combination. Not even slightly. In fact, they're a downright pain in the rear end. I've been bitten by this so many times recently that my rear end is raw.

If you haven't had the misfortune to encounter this particular brand of coding horror, consider the following simple C++ class. It creates a background thread which runs an arbitrary functor ("what") every "period" milliseconds:


class PeriodicCaller
{
public:
PeriodicCaller(const some_functor &what, unsigned period);
~PeriodicCaller();

private:
... whatever ...
};



It's got a nice, clean interface. It seems simple enough. It'll be really easy to use in your codebase. But how are we going to test it? Any ideas?

First problem: there are lifetime issues in that interface. "what" has to remain valid for as long as the background thread runs. That's hard to do, as you don't know exactly how long the thread runs for. Fortunately, you can solve this problem by insisting classes that spawn a background thread (or threads) guarantee the thread has stopped by the time the destructor completes. This moves the issue up a level: "what" has to remain valid for as long as the PeriodicCaller object exists. This is far, far easier to reason about (and not an unusual problem for C++ object construction).

OK, one down. What's next?

Unit tests stub out all peripheral code sections (using mocks or stubs) to provide a precisely controlled interface to test your unit of code. In this respect, unit tests rock - to write them you must ensure that your class connects only with a finite set of precisely defined interfaces. That's good design. Unit tests help to ensure your class design is sound. Cool.

In the tests, we use these interfaces to isolate the code under test from unexpected change, and to arrange sets of specific operational conditions for it the unit to run in.

Great stuff.

Threads add a new level of unexpected interactions and interactions. What tests should we write for the PeriodicCaller? Some of them might include:
  1. It calls the functor after "period" has elapsed
  2. It calls it N times after N "period"s have passed
  3. Once the PeriodicCaller is deleted, the functor is never called again (just how long should you wait to be sure it is never called again?)

These tests will be slow (they have to wait for specific "period"s - or arbitrary lengths of time - to elapse). It requires the test machine to not be too loaded (or the background thread might not get a chance to execute often enough, causing the test 2 to fail). You could only write these tests reliably with some control over the background thread.

In a unit test for PeriodicCaller, you can't control the background thread that is spawned. Even if there was a "for testing" accessor to the background thread object (returning a boost::thread, or whatever) how would you use that object to drive the thread to make unit testing predictable?

By definition, threads interweave in arbitrary ways. You simply can't guarantee you're covering all possible interactions of those threads in a unit test. Subtle thread interaction problems are a surefire recipe for unit tests that work most of the time, and collapse every so often. Hard to find, hard to fix.

You could try to inject sanity with an API over the background thread that allowed you to flatten the thready behaviour out and call it sequentially on the test thread. That might help avoid thread disasters in the unit test, but the test would not be reflecting the reality of threaded operation. All the problems would be masked, not removed.

It is specifically hard to unit test code that does any of these:
  • Spawns a new thread
  • Waits for a thread to finish
  • Synchronises with another thread

So what does this teach us? Should we avoid using threads? Well, no, clearly we can't do that - threads are useful. But since it's very, very hard to prove that our threaded code is correct, we should avoid writing any more threaded code than strictly necessary. And then we need to create interfaces to threaded components that are testable.

How can we craft testable thread interfaces? There are a few potential solutions. Before I consider writing about them, what do you think? How have you solved these problems?

Thursday 15 May 2008

Cross compiling Boost

Boost is a truly excellent C++ library, and something that all C++ programmers should be familiar with. It's a set of peer-reviewed extensions to the standard C++ library and is an excellent weapon in your C++ arsenal. Many core Boost libraries have gone on to be adopted in the C++ standard itself. I love it.

Mostly.

Building the Boost library is a swine. Seriously annoying. For most users, this isn't an issue. You can get binary versions for most platforms, and almost every Linux (or other Unix platform) distribution has a pre-packed version available. So that's fine. And to be fair, most of the Boost libraries are header-only (plenty of template mumbo-jumbo), so often it's irrelevant anyway.

But if you need to build Boost yourself you enter a weird world. Boost uses a homegrown variant of Perforce's Jam (which they call bjam - amusingly remeniscent of a 1980s UK white goods retailer). That's a little unusual, but fine. Then they layer their own build file magic on the top. That's fine. And then they document it. That's fine.

Apart from the bit where it's really hard to work out how to use it to do anything other than a simple build. For example, it's really, really hard to work out how to make the gcc build process use your own gcc compiler, rather than the standard gcc on $PATH. It's a real shame that such an excellent library requires you to become an expert in their non-standard build system in order to build it!

We run C++ on embedded ARM devices, so we cross compile to ARM from x86 Linux boxes. I recently needed to update our toolchain (from gcc 3.x to gcc 4), and I had to jump from Boost 1.33.x to 1.35 (the former had bugs when compiled with gcc 4). This newer Boost version saw subtle build system changes, and (IMVHO) a worsening of the build documentation.

In case you need to do the same thing - cross compile Boost with your own custom compiler - here's my recipe (this is for Boost 1.35, the process is annoyingly different for previous versions):

First you must make Boost jam. If you have a different jam it may not work. Specifically, if you want to use Cygwin you must use the correct jam from Boost not the Cygwin-supplied bjam, or things don't work in weird ways...

cd tool/src/jam
./build.sh
cp bin.*/bjam
PATH=:$PATH
cd -

Now, here's the magic. Brace yourself. This is how you use a custom gcc compiler, you write some odd rule into some very well hidden config file:

echo "using gcc : 4.2.2 : PATH_TO_DIR/arm-softfloat-linux-gnu-g++ ; " > tools/build/v2/user-config.jam

Obviously, tweak the version numbers to your requirements. Then use your fresh copy of jam to build Boost, changing the configuration magic below as you require...

bjam -d2 \
--toolset=gcc
'-sBUILD=release static multi/single' \
link=static \
--prefix= \
--layout=system \
--with-XXX --with-XXX \
install

And that should do the trick. (The --with-XXX lines specify which libraries you'd like built, e.g. "--with-thread --with-signals --with-filesystem")

Now, I'm not claiming that this recipe is the best way to do it. But from a lot of reading, it's the only half-way sane method that I've found. For example, it's possible to put symlinks to your custom toolchain early in your path, but that strikes me as a very clumsy way to persuade the Boost build system to use your compiler.

There is one problem thyat I have observed with this approach: it only uses your custom g++ compiler, but still uses the system ar. As it happens, this works fine (at least, it does for static libraries, which is what we use). Perhaps another using line in the randomly located jam config file might solve this?

That's my recipe. If you have to cross compile Boost then: good luck!

Tuesday 13 May 2008

Must-have Apple software

I used to use Linux as my primary platform. I'm now officially an Apple fanboy. I've had Macs for a while, but now I'm a total convert. The user experience rocks, and it's still Unix under there. And Cubase on the Mac works far, far better than anything on Linux.

There's loads of freeware available that's useful, but here's my shortlist of the genuine must-have Apple software, from an I-was-a-Linux-weenie point of view:
  • MacVim - The best port of (the best text editor) Vim to the Mac.
  • NeoOffice - Very good native port of OpenOffice, the multi-platform Microsoft Office replacement. OpenOffice will soon have a native Mac release, but until then this is great.
  • Adium - A multi-protocol instant messenger
  • Transmission - Sweet little torrent client
  • Audacity - Simple but capable audio editor
  • MacFUSE (and MacFusion) - Native filesystems for many protocols (ssh, NTFS, and many many more). MacFusion is a cute menubar GUI to make using MacFUSE more convenient.
  • The Unarchiver - Better than the built-in file decompression utility. Simple and unobtrusive.
  • The Gimp - The well-known image processing package. Sadly, runs under X11 - not a native app.
  • MacSword - An awesome interface to the Sword bible study package.
  • iRed Lite - Allows you to use the Apple remote to do all sorts of useful things (I've used it as a remote presentation controller)

Plugins and utilities:
  • Growl - Notification system for MacOS. Many applications support growl notifications.
  • Afloat - Useful program that allows most application's windows to be pinned to the top of the window stack. Watch a quicktime movie easily whilst working on a document!
  • Inquistor - Awesome Safari search plugin. Calls itself "Spotlight for the web". I can't argue with that,
  • MenuMeters - System profile utility that puts info up in the menu bars. Very useful to see how much CPU Cubase is eating up!
  • OpenTerminalHere - Great Finder plugin that opens a terminal window with the working directory set to the current finder path.
Commercial software I highly recommend:
  • Parallels - The infamous utility that lets you run Windows at the same time as Mac OS. It is truly awesome.
  • Cubase - Well, to be fair, there's a ton of muso application's I'd recommend, like the incredible TruePianos and many, many more.

Before leopard, there were other packages that were important to me:
  • iTerm - Tabbed terminal window. Now redundant as the Leopard terminal has been vastly improved.
  • Desktop Manager - Virtual desktop application. Now replaced by Leopard's built-in spaces feature. Interestingly, this application still plays very nicely with Leopard's spaces.

Friday 2 May 2008

ACCU Cambridge

Last night was the first ACCU Cambridge meeting that's been run a little while, and a definite return to form!

Roger Orr (from sunny Londonshire) took us on a tour of the joys, intricacies, and new toys coming with the forthcoming C++ "0x" language standard. As you'd expect from Roger, the talk was interesting and entertaining. The venue (again, the DisplayLink offices situated on the nearest thing to a hill we have in Cambridge) was packed, almost exclusively with C++ programmers who clearly wanted to know the new ways they will be able to write fascinating bugs once compiler vendors catch up with the ISO committee (because we all know what a breakneck speed the ISO standards are produced at).

The evening concluded with traditional aprรจs-talk beers at the Castle.

It was another excellent evening - many thanks to Roger for speaking and Ric for organising.