Building a Boost framework for iOS (iPhone, iPad, etc)

Several Boost versions have ticked past since I last posted about building a Boost framework for use with the iPhone. Each version of Boost requires subtly different build steps, and each version better supports building iPhone versions out of the box.

The good news: Boost 1.44.0 builds for iPhone a lot more easily than previous versions.

The bad news: making a framework is still not easy.

But first, why?

First, let's answer the obvious question: Why would you want to bother building a framework at all?

Quite simply: to make your life easier.

Using the native Boost build, you (may) need to link against a number of different Boost libraries in your application. And then you need to do a clever dance to ensure you link against either the i386 (simulator) or armv6/armv7 (iPhone devices) versions, depending on your build configuration.

That is tedious and annoying, and not the way it should be. Other Apple "libraries" are provided as a "framework" which you just need to mention once in Xcode, and then the build system works out how to link to the correct version and how to find the right headers.

So, if we can turn Boost into a framework life would be so much easier.

And that's what I have done.

Pete: Doing all the hard work to save literally... some... people all the hassle.

Executive summary

Now, you can do it too.

Just run the script you'll find in Gitorious here: http://gitorious.org/boostoniphone/boostoniphone

Download yourself boost_1_44_0.tbz into the same directory, run the script. Make a cuppa. Make another cuppa. And then admire your new shiney Boost framework.

Use it like any other framework. Draggy droppy. Happy happy.


What changed?

For the hardcore faithful who care about htese things, here's what's changed since my last script version:

• Boost now builds for iOS far more easily (I think this changed for the better in 1.43.0). Now, you just need to write some gibberish in a user-config.jam and run a boostrap script. That is a darned site easier than it used to be.

• To make it build for the simulator, you still need to invent some missing header files.

• But the biggest hassle in this version was creating the final universal library file. Since the ARM builds themselves are now fat archives (including an armv6 and armv7 version of each .o file) I needed to work out how to make a functional uber-lib file. Because Apple's binutils are not useful GNU ones, their ar is braindead. The only way to make a correctly functioning library (that I could find) was to un-lipo the fat ARM binaries, un-ar each archive, link all the original .os into an uber library, than lipo up each of those. Does that sound painful? Yes. It hurt.
  

Boost on the iPhone

This is the simple way to get Boost into your iPhone code.

I've been porting a large C++ project to the iPhone. It uses the excellent Boost libraries. Building Boost for the iPhone is not impossible, just a bit of a pain in the arse.

There are a number of good examples of how to do this online, for example the Backstage blog entry here and Matt Galloway's blog here. They are useful hints that help you work past the impenetrable Boost Build documentation.

However, the story does not end here. Those instructions allow you to build a set of libraries for the simulator, or for the iPhone OS. But not both. This means that your Xcode project setup gets fiddly with different link paths for the different targets.

You can solve this by creating a "universal" fat library. The lipo tool can be used to shunt the individual libraries together. Not tricky, just another step.

Now, for bonus points it would be sweet to construct a "Framework" for the Boost libraries, allowing you to use them in Xcode like any other iPhone framework. I've already blogged on how to do this here.

Of course, if you were sensible, you'd wrap this up in a script so that anyone can use it. A script a bit like this one.

I've set up a Gitorious project for this script. Feel free to use it.

C++: An STL-like circular buffer (Part 7)

Time to air the dirty laundry, and — since every good metaphor must run to the bitter end — to ask a pantomime washer woman to save the day. Or something like that.

In part 6 we identified some serious object management issues with the existing circular_buffer design. Now we understand them, we'll learn how to fix them, and how to use a C++ allocator. This whole sordid tale will show us exactly how useful allocators are.

What is an allocator?

All STL containers take an allocator as a template parameter. The allocator is an abstraction used to allocate and free memory for items held in the container. The allocator interface is described in section 20.1.5 of the C++ standard.

An allocator is itself templated over the type of object it can allocate space for. Every STL container defaults the allocator template parameter to std::allocator, the standard C++ allocator which employs new and delete. You'll rarely specify an allocator by hand in day-to-day C++ code. The std::allocator class is described in section 20.4.1 of the standard.

The allocator interface is pretty straightforward. An allocator provides a number of typedefs, identical to those required of a container:

• value_type
• pointer
• const_pointer
• reference
• const_reference
• size_type
• difference_type

It provides a number of methods, including:

size_type max_size(); // Same as a container

// Allocate memory. The second parameter is an optional "hint"
// to the allocator, the address of the object allocated prior to
// this allocation.
pointer allocate(size_type,
          typename std::allocator<void>::const_pointer = 0);

// Relinquish allocated memory
void deallocate(pointer p, size_type);

// Construct an object at this (allocated) memory location
void construct(pointer p, const T &t);

// Destruct the object that was constructed in this memory location
void destroy(pointer p);

allocate and deallocate, as their names suggest, service requests for memory allocation or deallocation. More interestingly, construct and destroy are used to vivify an object in already-allocated memory (the memory must have originally come from this allocator). The default implementations employ placement new and invoke the object's destructor in place, as below:

void construct(pointer p, const T &t) { new(p) T(t); }
void destroy(pointer p) { p->~T(); }

That's the kind of C++ you'll very, very rarely write, but that is worth understanding all the same.

Adding the allocator to circular_buffer

So let's thread an allocator through our circular_buffer. The first step, naturally, is to add a new template parameter to the class. We'll default this to std::allocator, which will suit 99% of all users.

template <typename T, typename A = std::allocator<T> >
class circular_buffer
{
    // ...

Since the allocator provides a host of useful typedefs related to the object it allocates, we can refine our class' definitions accordingly:

 typedef T                                        value_type;
typedef A                                        allocator_type;
typedef circular_buffer<T,A>                     self_type;
typedef typename allocator_type::difference_type difference_type;
typedef typename allocator_type::reference       reference;
typedef typename allocator_type::const_reference const_reference;
typedef typename allocator_type::pointer         pointer;
typedef typename allocator_type::const_pointer   const_pointer;

We also extend the constructor to take an explicit allocator object (and supply a sensible default), and add get_allocator to retrieve it.

explicit circular_buffer(size_type capacity,
                        const allocator_type &a = allocator_type());
allocator_type get_allocator() const;

We'll need to hold a copy of the allocator alongside the other private data members. But more importantly, we must use the it to allocate the memory for m_buffer and to delete it again afterwards. The boost::scoped_array (which seemed like such a good idea at the time) goes, and we must therefore add a corresponding custom destructor, too.

public:
  ~circular_buffer();

private:
  size_type      m_capacity;
  allocator_type m_allocator;
  pointer        m_buffer;
  pointer        m_front;
  pointer        m_back;

The updated method implementations look like this:

template <typename T, typename A>
inline
circular_buffer<T,A>::circular_buffer
    (size_type capacity, const allocator_type &allocator)
: m_capacity(capacity),
m_allocator(allocator),
m_buffer(m_allocator.allocate(capacity)),
m_front(0),
m_back(m_buffer)
{
}

template <typename T, typename A>
inline
circular_buffer<T,A>::~circular_buffer()
{
  clear(); // Delete all objects before deallocating the buffer
  m_allocator.deallocate(m_buffer, m_capacity);
}

template <typename T, typename A>
typename circular_buffer<T,A>::allocator_type
inline
circular_buffer<T,A>::get_allocator() const
{
  return m_allocator;
}

template <typename T, typename A>
inline
typename circular_buffer<T,A>::size_type circular_buffer<T,A>::max_size() const
{
  // This is clearly more elegant than the previous implementation!
  return m_allocator.max_size();
}

// This version of push_back must constuct and detroy objects in m_buffer
// using m_allocators methods, rather than traditional construction, copying,
// assignment.
template <typename T, typename A>
inline
bool circular_buffer<T,A>::push_back(const value_type &value)
{
  // If the buffer is full, and data will fall off the back of the
  // buffer - so we must destroy the stale object first.
  // This is an implementation choice, we could instead use operator= on
  // the value_type to replace the item. However, that would require
  // value_type to provide a copy assignment operator.
  if (m_front && m_front == m_back)
      m_allocator.destroy(m_back);

  m_allocator.construct(m_back, value);

  // The rest of push_back is as it was before
  value_type *const next = wrap(m_back+1);
  if (!m_front)
  {
      // first entry in the buffer
      m_front = m_back;
      m_back = next;
      return true;
  }
  else if (m_front == m_back)
  {
      // buffer is full already, throw something away
      m_front = m_back = next;
      return false;
  }
  else
  {
      m_back = next;
      return true;
  }
}

template <typename T, typename A>
inline
void circular_buffer<T,A>::pop_front()
{
  assert(m_front);

  m_allocator.destroy(m_front);
  value_type *const next = wrap(m_front+1);
  if (next == m_back)
      m_front = 0;
  else
      m_front = next;
}

template <typename T, typename A>
inline
void circular_buffer<T,A>::clear()
{
  if (m_front)
  {
      do
      {
          m_allocator.destroy(m_front);
          m_front = wrap(m_front+1);
      }
      while (m_front != m_back);
  }
  m_front = 0;
}

That's everything that needs substantial change. Every other existing method definition gains an additional template parameter (but ignores it). Oh, and wrap() needs the ".get()"s striped from the m_buffer address lookups.

That's it.

We've fixed some nasty, subtle bugs and taken one step closer to STL perfection.

Next time: We begin to iterate. And then we iterate. And then we iterate. And then we...

The trials and tribulations of Boost 1.36.0

I've been updating the toolchain that we use to build our products recently; we've upgraded to a much newer Linux kernel and use an (almost) up-to-date gcc that generates much better ARM code. Woo hoo!

Whilst I was at it, I decided to upgrade some of the third party libraries we depend on. I updated to Boost 1.35.0 (which was the latest version available when I started), and got the ARM build working successfully (some of our source code had to be modified for the new library version, mostly around boost::filesystem and boost::thread).

Since Boost 1.36.0 has recently been released, I thought it would be worth moving up another 0.1 worth of code and using that before I took the new toolchain live. It aparently has bugfixes around the threading code. Sounds useful. But oh, the best-laid plans of mice and men often go awry...

Sigh.

I have suffered two nasty Boost-related problemettes that I will share with you, gentle reader. One, to be fair is not just 1.36.0's fault...

1. shared_ptr with posix threads locks up on x86 platforms

It's easy enough to say it, but it took me a while to work out what was going wrong.

Since we target ARM devices, and ARMs do not have an atomic increment/fetch (which boost::shared_ptr relies on) we have to build it with a Posix thread library shared_count backend (by forcing -DBOOST_SP_USE_PTHREADS through to the compile using evil bjam config foo).

We also run our code on local x86 development machines for convenience (and to run our unit tests locally). To ensure the execution environment is as similar as on the target machine, I've always configured Boost to use posix locks around shared_count on this platform, too. It made sense. And it worked fine on 1.34.x versions.

However, on 1.36.o (and, as it happens, on 1.35.0, too - but I only discovered that later), that combination does not work. At least with gcc 4.2.2 and gcc 4.2.3.

Any boost::thread object you create fails to start, and wedges the calling thread. (Internally, the boost::thread::start_thread method in the posix implementation attempts to assign a shared_ptr variable, which causes a deadlock around the shared_count's pthread_mutex_lock call. I don't understand how or why that would fail; there appears to be no way that mutex would be used elsewhere. But there it is; it locks up. I am wondering about random comsic rays or, more likely, a compiler bug: If you disable compiler optimisations the deadlock magically disappears (which makes stepping through the code in gdb to find out what the problem is... tricky).

Solution #1: don't configure boost with -DBOOST_SP_USE_PTHREADS on Intel machines.

2. ARM builds of boost 1.36.0 will not link.

Blasted thing. I finally got the codebase to compile against the 1.36.0 verison of boost and would it link? No it would not. It gave up with lots of bitching about __sync_add_and_fetch_4. This is a glibc internal function that is not supported on ARM platforms (the ARM instruction set does not make such an operation supportable).

Now, I've stared at this problem for a reasonable length of time, and I can't actually see which bit of the Boost codebase is (directly or indirectly) pulling in a reference to this symbol. But it is. And it shouldn't be. For the time being, this problem has beaten me.

At the moment, I have to get the toolchain live rather than waste more precious developer hours, so I've regressed back to Boost 1.35.0 which does not suffer this linkage problem.

Solution #2: Do not use boost 1.36.0. (Yet.)

Sigh.

I hope this whittering blog entry will help other people who get stuck in similar predicaments.

Cross compiling Boost

Boost is a truly excellent C++ library, and something that all C++ programmers should be familiar with. It's a set of peer-reviewed extensions to the standard C++ library and is an excellent weapon in your C++ arsenal. Many core Boost libraries have gone on to be adopted in the C++ standard itself. I love it.

Mostly.

Building the Boost library is a swine. Seriously annoying. For most users, this isn't an issue. You can get binary versions for most platforms, and almost every Linux (or other Unix platform) distribution has a pre-packed version available. So that's fine. And to be fair, most of the Boost libraries are header-only (plenty of template mumbo-jumbo), so often it's irrelevant anyway.

But if you need to build Boost yourself you enter a weird world. Boost uses a homegrown variant of Perforce's Jam (which they call bjam - amusingly remeniscent of a 1980s UK white goods retailer). That's a little unusual, but fine. Then they layer their own build file magic on the top. That's fine. And then they document it. That's fine.

Apart from the bit where it's really hard to work out how to use it to do anything other than a simple build. For example, it's really, really hard to work out how to make the gcc build process use your own gcc compiler, rather than the standard gcc on $PATH. It's a real shame that such an excellent library requires you to become an expert in their non-standard build system in order to build it!

We run C++ on embedded ARM devices, so we cross compile to ARM from x86 Linux boxes. I recently needed to update our toolchain (from gcc 3.x to gcc 4), and I had to jump from Boost 1.33.x to 1.35 (the former had bugs when compiled with gcc 4). This newer Boost version saw subtle build system changes, and (IMVHO) a worsening of the build documentation.

In case you need to do the same thing - cross compile Boost with your own custom compiler - here's my recipe (this is for Boost 1.35, the process is annoyingly different for previous versions):

First you must make Boost jam. If you have a different jam it may not work. Specifically, if you want to use Cygwin you must use the correct jam from Boost not the Cygwin-supplied bjam, or things don't work in weird ways...

cd tool/src/jam
./build.sh
cp bin.*/bjam
PATH=:$PATH
cd -

Now, here's the magic. Brace yourself. This is how you use a custom gcc compiler, you write some odd rule into some very well hidden config file:

echo "using gcc : 4.2.2 : PATH_TO_DIR/arm-softfloat-linux-gnu-g++ ; " > tools/build/v2/user-config.jam

Obviously, tweak the version numbers to your requirements. Then use your fresh copy of jam to build Boost, changing the configuration magic below as you require...

bjam -d2 \
--toolset=gcc
'-sBUILD=release static multi/single' \
link=static \
--prefix= \
--layout=system \
--with-XXX --with-XXX \
install

And that should do the trick. (The --with-XXX lines specify which libraries you'd like built, e.g. "--with-thread --with-signals --with-filesystem")

Now, I'm not claiming that this recipe is the best way to do it. But from a lot of reading, it's the only half-way sane method that I've found. For example, it's possible to put symlinks to your custom toolchain early in your path, but that strikes me as a very clumsy way to persuade the Boost build system to use your compiler.

There is one problem thyat I have observed with this approach: it only uses your custom g++ compiler, but still uses the system ar. As it happens, this works fine (at least, it does for static libraries, which is what we use). Perhaps another using line in the randomly located jam config file might solve this?

That's my recipe. If you have to cross compile Boost then: good luck!