First, a quick recap of reasons why we don’t want to use pointers for referencing data is in order:

• With raw pointers, ownership is sometimes unclear. If I’m handed a pointer, do I need to delete the instance? Who owns it? How long can I hold on to it?
  This very quickly leads to double-deletes and/or dangling pointers. Both are kinds of bugs which can be hard to find if you’re unlucky.
• The above can be somewhat alleviated by using a shared_ptr<> or some reference-counting mechanism, but now we have added additional overhead which isn’t really necessary. Ownership is still unclear – when is the data behind the reference-counted pointer actually freed? Who else holds on to it?
• How are pointers replicated or copied, e.g. across the network? You always have to have some kind of serialization mechanism in place because you cannot just send pointers across the network – the address they contain won’t make sense in a different address space.
• Pointers don’t support relocation. Ultimately, the system who owns the data should also be responsible for managing the data’s memory. Therefore, a system might want to move things around in memory, e.g. for run-time defragmentation. Notifying each and every instance that might hold a pointer to system-internal data is tedious and error-prone.

So, let us now take a closer look at how we can store internal references without running into the above-mentioned problems.

Handles

In the Molecule Engine, handles are used to refer to internal data. That is, they refer to data owned by some system directly, and not via some indirection. This is also the reason why they are called internal references.

What are handles? Basically, they are indices into the data, but with a twist. One can think of handles as „smart indices“. But before going into detail about handles, which problems do plain indices already solve?

• You cannot accidentally call delete or free() on an index. Furthermore, if a system only deals with indices as input and output parameters, it should be clear that the system also owns the data.
• Indices can be easily replicated and copied. They also support data relocation out-of-the-box: if we want to access the data at e.g. index 3, it doesn’t matter where the data itself resides, as long as it remains in the same order. It can reside at address 0xA000 or 0xB000 or someplace else – data[3] will give us the data we want.

Of course, there are things which are not supported by plain indices:

• We cannot detect access to stale/deleted data. We might try to access the data at index 3, but that might have been freed already since our last access.
• Whole data blocks can be moved around in memory, but the order of individual data items cannot be changed, because that would mess up our indices.

Handles help us with the first problem, but also don’t support arbitrary relocation of individual data items. This is what IDs or external references are for, but those will be the topic of the next post.

The question remains: how do we turn indices into handles that can detect access to already deleted data?

The idea is quite simple: instead of only using an index, a handle also stores the generation in which the index was created. The generation is simply a monotonically increasing counter that gets incremented each time a data item is deleted. The generation is stored both inside the handle, and for each data item. Whenever we want to access data using a handle, the index’ generation and the data item’s generation need to match.

An example

Going back to our example from the last post, let us assume our render backend provides space for 4k vertex buffers. New vertex buffers are allocated using a pool-allocator/free-list internally, and users only deal with a VertexBufferHandle.

Initially, our pool of vertex buffers is empty, and all generations are set to zero.

4096 Vertex buffers:
+----+----+----+----+----+----+
| VB | VB | VB | .. | VB | VB |
+----+----+----+----+----+----+

4096 Generations:
+----+----+----+----+----+----+
| 0  | 0  | 0  | .. | 0  | 0  |
+----+----+----+----+----+----+

The first time we allocate a vertex buffer, the handle will contain an index of 0, and a generation of 0. Future vertex buffer handles will have a different index, and also a generation of 0.
Assume we now destroy the first vertex buffer we allocated. The generation of the slot that contained it will increment, yielding the following layout:

+----+----+----+----+----+----+
| VB | VB | VB | .. | VB | VB |
+----+----+----+----+----+----+
+----+----+----+----+----+----+
| 1  | 0  | 0  | .. | 0  | 0  |
+----+----+----+----+----+----+

If we now want to access that vertex buffer using the handle, we check its generation against the one stored with our vertex buffer, and find that they don’t match – meaning that we tried to access already deleted data.

In code, this situation looks somewhat like the following:

VertexBufferHandle handle = backend::CreateVertexBuffer(...);

// some more vertex buffers created in the meantime

// at a later point in time, we destroy the vertex buffer...
backend::DestroyVertexBuffer(handle);
// ...but somebody, somewhere, still holds the same handle

backend::AccessVertexBuffer(handle);  

Handle implementations

One thing we haven’t talked about yet is how handles can be implemented. As almost always, the simplest solutions are the best, so a trivial struct will suffice in this case:

struct Handle
{
  uint32_t index;
  uint32_t generation;
};

In practice, you normally would not use two 32-bit integers for both the index and the generation, but rather use bitfields instead. In the case of our vertex buffer handles, we need 12 bits for storing indices in the range [0, 4095], which leaves 20 bits for the generation if we want our handles to be 32-bit integers. Hence, our handles would look more like the following:

struct Handle
{
  uint32_t index : 12;
  uint32_t generation : 20;
};

This means that the generation overflows after 1048576 vertex buffers have been deleted in the same slot in our pool. Theoretically, this means that we could wrongly access a vertex buffer via an old handle that was generated more than 1048576 vertex buffer create/delete cycles ago, in that very slot. In practice this should never happen, unless we store an old handle for ages, create/delete buffers like crazy, and never access the buffer using that handle in the meantime.
Yet, depending on the number of bits you are willing to spend this can happen, so it is something to keep in mind.

Last but not least, another nice thing about handles that I mentioned in the previous blog post is that they use less memory than a pointer. Most handles can store their index and generation in a single 32-bit integer, which means they need half the amount of memory compared to 64-bit pointers. Additionally, we really only need to store the generation inside a handle for detecting access to stale data. We should not need that in a retail build, hence handles can be as small as 16-bit integers in those builds, if your indices only need to be in the range [0, 65535].

A generic implementation

In Molecule, I use a generic handle implementation which defines the underlying data types according to certain build rules, and also static_asserts whether the bits fit into that type. The basic struct is as follows:

template <size_t N1, size_t N2>
struct GenericHandle
{
  // uint16_t or uint32_t, depending on build type, realized using preprocessor-#ifs
  uint32_t index : N1;
  uint32_t generation : N2;
};

All handle types then become simple typedefs, e.g.:

typedef GenericHandle<12, 20> VertexBufferHandle;

And that concludes today’s post! In the next part in the series, we will discuss how external references also allow for moving individual data items around in memory, without user code having to care about that.

Defining ownership initially requires people to think more about who owns data, who creates and destroys instances, but totally pays off in terms of maintenance, performance, and debuggability. I would like to go back to one of my favourite examples, because it is easily understood by everyone: rendering a bunch of static meshes.

Mesh rendering

The example we are looking at is the following:

• A level contains any number of static meshes. A static mesh consists of a vertex buffer, an index buffer, and a bunch of triangle groups describing the indices used by each group/submesh in the mesh. We call a struct/class holding that information a Mesh.
• Each of these meshes can be rendered with a different shader and material, hence those are not part of the mesh, but are used by something called a MeshInstance, a MeshComponent, or similar. We need this distinction because the same mesh can be instantiated with different shaders and materials several times in a level.

If you do not care much about memory, performance, and keeping your co-workers sane, the easiest solution is to allocate each Mesh individually using new, and store a pointer to a Mesh inside the MeshInstance. Because several such instances can now refer to the same Mesh, you either hold a shared_ptr<Mesh> or similar, or build some other means of reference-counting into the Mesh class and store a plain old raw pointer. And while you’re at it, vertex buffers and index buffers are also reference-counted, because you can and it’s nice and OOP-esque. Problem solved.

Well, yes, and no.

There are several major disadvantages with using such an approach:

• Ownership: Who owns instances of Mesh? Whenever a MeshInstance is deleted or goes out-of-scope, it simply decrements the reference-count of the referenced object. Can anybody tell when the actual object gets destroyed?
  The same is true for vertex buffers and index buffers. When are those deleted, exactly?
• Performance: You’re handing out raw pointers, shared_ptr<> or anything along those lines to users of your class. This makes it very hard (if not impossible) to move things around in memory, and still let all pointers point to the correct object. Hence, most of your objects are scattered all over the heap, causing tons of cache-misses upon accessing them, because subsystems cannot rearrange objects in memory as they best see fit.
• Debuggability: Raw pointers just scream „dangling pointer“. Oh, you correctly released a Mesh reference, the original object got destroyed, but you are still holding on to your MeshInstance? Well, the memory manager allocated a new instance at the exact same spot in memory in the meantime, and you’re now working with stale data without noticing. Guess it’s not your lucky day.

Of course we can do better.

A better solution

The first thing to think about is: who uses the data, who owns, creates and destroys it? Let’s look at vertex and index buffers first.

The only thing that should ever be doing API calls in a rendering engine is the render backend. In Molecule parlance, the rendering backend is just a namespace with tons of free functions, who directly talk to D3D11, OpenGL, DX9, or other APIs. Porting the graphics module means porting the backend, and additionally exposing platform-specific functionality, but that’s about it.

The backend is also responsible for queueing and sorting draw calls according to 64-bit keys, and hence is the only thing in the engine binding vertex buffers, setting render states, performing actual draw calls, etc.

Because the backend is the only thing touching that data, it would benefit the most if all low-level rendering-related data such as vertex and index buffers were as close together in memory as possible. Therefore, the backend itself should also be responsible for creating/destroying those buffers, taking ownership over them.

In addition, we do not want to return raw pointers to our internals, because we want to be able to track accesses to stale data – no more dangling pointers! Furthermore, by giving the user a simple identifier such as an integer instead of a pointer, the question of „Do I own this? Do I need to free this?“ never actually arises.

Simplicity trumps everything

So what is the simplest solution for getting something that is as close together as possible in memory? An array.

This is what Molecule uses. The rendering backend simply holds an array of 4096 vertex and 4096 index buffers. Of course those numbers are configurable, but do you ever need more than 4k vertex buffers in flight at the same time? If so, you have a worst-case scenario of at least 4k distinct draw calls in a certain frame, which is unreasonable anyway (in terms of performance).

Instead of a pointer, you can now simply return a 16-bit integer that can be used to uniquely identify vertex and index buffers in your array – it is nothing more than an index into the array. Not only is the question of ownership no longer a question (you cannot delete or free() a 16-bit integer, nor decrement its reference-count), you can also build in a mechanism for tracking whether a given integer refers to an existing object or not – this is what is often referred to as a handle. Depending on the maximum number of instances, a 16-bit int might suffice, or you can always go to 32-bit.

That being said, the interface for creating and destroying vertex and index buffers in Molecule looks like this:

namespace backend
{
  VertexBufferHandle CreateVertexBuffer(unsigned int vertexCount, unsigned int stride, const void* initialData);
  VertexBufferHandle CreateDynamicVertexBuffer(unsigned int vertexCount, unsigned int stride);
  void DestroyVertexBuffer(VertexBufferHandle handle);

  IndexBufferHandle CreateIndexBuffer(unsigned int indexCount, IndexBuffer::Format::Enum format, const void* initialData);
  void DestroyIndexBuffer(IndexBufferHandle handle);
}

Referencing mesh data

Thinking about mesh data ownership, we can come up with a similarly simple solution for referencing/storing that data as well.

In the Molecule Engine, a thing called the render world holds all data which is tied to the graphics module, the main things being stuff that is pulled in from resource packages, such as meshes, skeletons, animations, particle systems, graphics-related components, etc.

Similar to the fixed-size vertex and index buffers that are being held by the render backend, the render world stores e.g. an array of all meshes contained in a resource package. Because all other rendering-related data is also owned by the render world, we can reference that by using handles as well.

This means that a Mesh now looks like this:

struct Mesh
{
  VertexBufferHandle m_vertexBuffer;
  IndexBufferHandle m_indexBuffer;
  TriangleGroupHandle m_triangleGroups;
  uint16_t m_triangleGroupCount;
};

No reference-counting, no shared_ptr<>, no raw pointers. A Mesh is trivially copyable, and can be moved around in memory by using memcpy(). How do you hold on to a Mesh? What does a MeshInstance look like?

It’s simple: you just copy the Mesh. You hold on to a Mesh simply by copying it. A MeshComponent just stores a copy of Mesh, along with handles for shaders, materials, and so on.

In practice, MeshComponents themselves are owned by the render system which is responsible for rendering them, but that is something for another blog post.

Conclusion

Let us quickly recap:

• Vertex and index buffers are owned by the render backend. No raw pointers are handed out, only handles. Handles are an opaque data type, and the user should not (and does not) know how to interpret the given integer.
• Mesh instances are owned by the render world. Meshes are referenced simply by copying them, because that gives you all the data you need in order to do something with it.
• There are no reference-counting mechanisms, no raw pointers, and most importantly no dangling pointers. The system automatically identifies accesses to stale data. In addition, most handles occupy less memory than pointers, especially on 64-bit systems.
• Mesh instances, MeshComponents, and many other components are merely data containers, and as such can be freely moved around in memory, without having to worry about ownership, construction/deletion, etc.

In the next posts, we will take a closer look at what Molecule uses for referencing data that is moved around in memory by subsystems responsible for updating/rendering it. One such system is the one responsible for rendering meshes, where it’s crucial that data is accessed in a cache-friendly fashion. Specifically, we will go into detail about internal references (= handles), and external references (= IDs).

Check out www.molecular-matters.com for more information on the input library. Further SDKs will follow during the next few months.

Specifically, we will look at the difference between a simple OOP-based by-the-book implementation, and a more data-oriented design. We are going to cover the differences both in terms of implementation and performance. And once more we will see that data-oriented design and object-oriented programming do not contradict each other!

Skeletal animation

Before we can start thinking of a possible implementation, we need to cover some ground first by quickly identifying the steps needed for skeletal animation.

Typically, an animated mesh such as a character consists of several joints that are connected with each other in a hierarchical fashion. The animation data specifies animation curves for each of the different components that make up the transformation of a joint, e.g. x-, y- and z-values for translation (and scaling, if supported), and x-, y-, z- and w-values for the rotation (stored as a quaternion).

Skeleton consisting of 96 joints

The data of the animation curves of each joint is stored in their local frame (= their local coordinate system) because that is easier to animate, and leads to more natural-looking animations. This means that whenever you sample the animation curves, all the transformations are stored in the local coordinate frame, and need to be transformed into the global coordinate frame in order to render the joints or the skinned character. Hence, sampling the animation curves gives us a transformation for each joint, called the local pose.

Note that if we want to blend between different joint transformations (typically done using LERP or SLERP), those blends are performed on local poses. After blending, layering, etc. has finished, all joints are transformed into their global pose. This is done using the joint hierarchy.

To recap, these are the steps that need to be performed:

1. Work out at what time we want to sample the animation curves. This is influenced by the speed of the animation, whether it’s looping, etc. The result of this is called the local pose.
2. Optionally blend between different local poses to smooth the animation.
3. Transform the local pose into the global pose according to the hierarchy.

I deliberately left out other steps such as ragdolls, steps that can affect the global pose and need a round-trip to the local pose (IK, FK), and others, in order to focus on the key parts of this post.

Demo setup

The demo we will be using to measure the difference between different implementations simply performs the above steps for 1000 characters. Our timings will measure how long it takes to perform step 3 (traversing the hierarchy and transforming the joints’ poses) a thousand times. We will use the character consisting of 96 joints (shown above) for this demo.

An army of 1000 characters, all sampled at a different point in time

Initial implementation

A simple, naive implementation of a hierarchical, skeletal data structure could look like the following:

struct Joint
{
  math::matrix4x4_t globalPose;
  math::matrix4x4_t localPose;
  std::vector<Joint*> children;
};

Simple enough: each joint stores a local pose, a global pose, and an array of child joints. A skeleton then becomes an array of joints, and that’s it. In order to transform the skeleton starting from the root, a simple recursive function can be used:

class Skeleton
{
public:
  void LocalToGlobalPose(void)
  {
    // start at the root
    LocalToGlobalPose(&joints[0], math::MatrixIdentity());
  }

private:
  void LocalToGlobalPose(Joint* joint, math::matrix4x4_arg_t parentTransform)
  {
    joint->globalPose = math::MatrixMul(parentTransform, joint->localPose);

    // propagate the transformation to the children
    for (size_t i=0; i < joint->children.size(); ++i)
    {
      LocalToGlobalPose(joint->children[i], joint->globalPose);
    }
  }

  Joint* joints;
};

The above traverses the hierarchy starting from the root, and propagates the transformation of a parent joint down its children. This is done by first storing the resulting global pose of the joint, and then passing it along to the children. The code is short, and should be pretty self-explanatory.

So the question is, how long does it take to build the global pose for 1000 characters using the above code? The answer: 3.8ms. We cannot decide yet whether that’s good or bad, but we can surely do better, otherwise I wouldn’t post about it.

A data-oriented approach

The first thing to observe here is simple, and should not come as a surprise to most of you: instead of storing the children for each joint, we can turn the problem on its head, and just store a parent joint for each joint instead. So instead of storing a std::vector<Joint*> children we store a Joint* parent instead.

The next optimization is also simple: we store all joints of a skeleton in an array anyway, so why not store indices instead of pointers? Assuming no skeleton has more than 65536 joints (which is a safe assumption I would say), we can store a uint16_t parent instead of the pointer previously mentioned.

The benefits?

• Storing an uint16_t instead of a pointer needs less space, especially on 64-bit systems. We save memory, and have less memory to access when building the global pose.
• A complete skeleton is now trivially copy-able, and can be moved around in memory using e.g. memcpy. Similarly, a whole skeleton can be loaded from disk using a single binary read without having to worry about pointer-fixups or similar. This alone is worth the data transformation.

The question that remains is: how do we change the code so that we still traverse the hierarchy in the proper order, and how do we get rid of the recursion?

In order to fix that, there is one key observation to make: as long as we always traverse the hierarchy in the same order, we can allocate and store the joints in that exact order, and walk through the array of joints linearly by flattening the hierarchy. Looking at the recursive code above, we can see that the code traverses the hierarchy in what is called depth-first order.
Now imagine traversing the hierarchy, numbering the joints that are visited with a monotonically increasing number. Joint 0 would be the root, joint 1 would be the next in the hierarchy following a depth-first traversal, and so on. This means that if we iterate through the array of joints, a joint i can only have a joint j as its parent with j < i. This greatly simplifies the traversal code, and gets rid of the recursion.

One last thing left to do is change the layout from AoS (array-of-structures) to SoA (structure-of-arrays), leaving us with the following:

class Skeleton
{
private:
  uint16_t* hierarchy;
  math::matrix4x4_t* localPoses;
  math::matrix4x4_t* globalPoses;
};

This change has the benefit that whenever we access e.g. the parent of a joint using hierarchy[i], several dozens of the next parents will also be read into the cache with the same cache-line. The same is true for accessing localPoses[i].
So what does the transformation code look like? It’s become surprisingly simple:

void LocalToGlobalPose(void)
{
  // the root has no parent
  globalPoses[0] = localPoses[0];

  for (unsigned int i=1; i < NUM_JOINTS; ++i)
  {
    const uint16_t parentJoint = hierarchy[i];
    globalPoses[i] = math::MatrixMul(globalPoses[parentJoint], localPoses[i]);
  }
}

Beautiful, and doesn’t get any simpler.

As stated above, this works because parentJoint < i, which means we will only access global poses which have been computed already. The recursive traversal that we had before is gone and is now implicitly given by the positions of joints in the array, which we numbered in a depth-first fashion.

What difference in performance does this make? With the same amount of transformations, this code now takes 3.1ms. Compared to the 3.8ms we had, that’s more than an 18% speed increase!

Conclusion

Not only did our few simple changes result in a nice increase in performance, it also vastly simplified the code itself. And we didn’t have to abandon OOP principles, we didn’t even have to change the calling code. We still have our skeleton class, and call LocalToGlobalPose() on it.

Concise, fast, and simple.

As explained in one of the last posts, virtual memory allows us to reserve address space without allocating any physical memory for it. We can use this to our benefit, and build an allocator that reserves address space up-front, and allocates physical memory whenever we need it. In that regard, the allocator behaves similar to the non-growing stack allocator, but is not restricted to working within a fixed region of memory.

Such an allocator is extremely useful for situations like e.g. loading level data, where the amount of memory needed varies between different levels. Using a growing allocator saves development time because there is no need to constantly change the upper bound of a fixed-size allocator whenever a certain level exceeds the previous worst-case memory usage.

An example

Consider a game where the maximum amount of memory that is spent on level-resident resources is defined to be 128 MB. What we want to have is an allocator that reserves address space for e.g. 256 MB (to be on the safe side), and allocates physical memory in 1 MB blocks whenever needed.

The basic steps of building such an allocator are simple:

• Reserve address space for 256 MB.
• Store the start and end of the reserved address space.
• Store the start and end of the physical address space (=0 bytes in the beginning).
• For each allocation:
  • Work out the required amount of physical memory.
  • If there is still enough physical memory left, bump the start of the physical address space (see above).
  • If there is not enough memory left, allocate 1 MB pages at the end of the current physical address space until the allocation fits.
  • Update the start and end of the physical address space.

In code this could look like the following:

GrowingStackAllocator::GrowingStackAllocator(unsigned int maxSizeInBytes, unsigned int growSize)
  : m_virtualStart(virtualMemory::ReserveAddressSpace(maxSizeInBytes))
  , m_virtualEnd(m_virtualStart + maxSizeInBytes)
  , m_physicalCurrent(m_virtualStart)
  , m_physicalEnd(m_virtualStart)
  , m_growSize(growSize)
{
}

In the constructor, we just reserve address space and store the start and end of it.

Allocating memory

void* GrowingStackAllocator::Allocate(size_t size, size_t alignment, size_t offset)
{
  const size_t allocationSize = size;

  // work out proper allocation sizes and possible offsets
  ...

  m_physicalCurrent = core::pointerUtil::AlignTop(m_physicalCurrent + offset, alignment) - offset;

  // is there enough physical memory left?
  if (m_physicalCurrent + size > m_physicalEnd)
  {
    // out of physical memory. check if there is still address space left from which new physical pages can be allocated.
    // remember that virtual memory must always be allocated in page-size chunks, that is why we round the needed size to
    // the next grow-size multiple.
    const size_t neededPhysicalSize = bitUtil::RoundUpToMultiple(size, m_growSize);
    if (m_physicalEnd + neededPhysicalSize > m_virtualEnd)
    {
      // the allocation doesn't fit into the address space, we're out of memory
      return nullptr;
    }

    // allocate new memory pages at the end of our currently allocated pages
    virtualMemory::AllocatePhysicalMemory(m_physicalEnd, neededPhysicalSize);
    m_physicalEnd += neededPhysicalSize;
  }

  // store book-keeping information if necessary
  ...

  // return allocation to user
  return m_physicalCurrent;
}

As described above, we allocate new pages whenever there is not enough physical memory left for the allocation to fit into. In that case, new pages are allocated at the end of the address range that already has physical memory committed to it. If there is not enough space left in the address space dedicated to this allocator, then we are out of memory.

The granularity with which the allocator grows is specified by the user in the constructor. It is a typical performance/memory-tradeoff regarding performance of allocations vs. wasted memory. If the allocator grows in 64 KB pages, it has to do lots of small allocations via the virtual memory system, but only wastes up to 64 KB, which is not much. In comparison, growing in 1 MB pages leads to far fewer allocations, but can waste up to 1 MB of memory that is backed by physical memory, but never used.

Freeing memory

One thing that is left to discuss is how the allocator behaves when freeing allocations. The primary thing to consider is whether the allocator should return physical memory pages whenever enough allocations have been freed. Depending on the allocation behaviour, that can be either good or bad.

When freeing level-resident resources, we mostly have several large allocations, and want to release physical memory as soon as possible. In situations where the allocator is used to make many small allocations, it could be more beneficial to keep physical memory that is committed to the address space. Otherwise, the allocator could suffer performance penalties whenever allocations straddle a page boundary, causing pages to be constantly allocated, freed, allocated again, and so on.

The way this is handled in Molecule is that whenever an allocations is freed, the allocator does not return physical memory to the OS. Instead, physical memory which is no longer
needed must be explicitly released by calling Purge(), which is an extra method offered by growing allocators. In case of the growing stack allocator, it simply checks which
pages are no longer needed, and returns them to the OS:

void GrowingStackAllocator::Purge(void)
{
  // we need to free all physical memory pages which are no longer needed by any allocation while making sure that we don't free the page
  // we're currently pointing to (remember that virtual memory only works in page-size granularity).
  // we do this by rounding the current physical memory pointer to the next grow-size boundary, and freeing all physical memory from there.
  char* addressToFree = pointerUtil::AlignTop(m_physicalCurrent, m_growSize);
  const unsigned int sizeToFree = safe_static_cast(m_physicalEnd - addressToFree);
  virtualMemory::FreePhysicalMemory(addressToFree, sizeToFree);

  m_physicalEnd = addressToFree;
}

Conclusion

Having growing allocators that make use of the virtual memory system allows us to specify a safe upper-bound for the amount of memory needed during development without wasting physical memory. Furthermore, knowing in which address range certain allocations lie can be a tremendous help when debugging.

Master class: Memory Management Strategies (PPT, PDF).

Session: Debugging memory stomps and other atrocities (PPT, PDF).

A big “Thank you!” to all the people who attended, I really enjoyed working with you. Looking forward to seeing some of you again next year!

Check out www.molecular-matters.com for more information on the core library. Further SDKs will follow during the next few months.

I’ll also be giving a talk about debugging memory stomps on the following day.

It would be a pleasure to meet up with some of you there!

What is virtual memory?

Speaking in simple terms, virtual memory provides an extra indirection when accessing memory. It provides an abstraction of the virtual address space of a process, which lets each process think that it’s alone in the system. It lets us write programs without having to worry which other programs allocated memory, and without having to worry which physical memory we need to access. Even though virtual memory is often also mentioned in conjunction with paging to hard disk (=swapping), this is not what we are interested in!

Consider a system having 4 GB of RAM that consists of four physical 1 GB RAM units. We are able to allocate more than 1 GB of contiguous memory, even though there’s no actual physical memory unit that is larger than 1 GB. This works thanks to virtual memory.
Similarly, the allocations we make inside an application return virtual addresses which are valid inside our process. Such an address could be 0×80000000, and another process can also have allocations residing at 0×80000000, and yet everything works thanks to virtual memory.

Address translation

Before touching actual physical memory, the virtual address needs to be translated. This virtual address to physical address translation is being taken care of by the MMU of the CPU. Modern CPUs also have a TLB which is used to speed up this translation.

Traditionally, this translation is done on a page-by-page basis. This means that on the OS-level, memory can only be allocated in so-called pages of a certain size. As an example, Windows 7 has a default page-size of 4 KB. Consequently, this also means that whenever you allocate memory directly from the OS, you can only allocate it with page-size granularity.

The details of address translation are actually quite involved, and here is a very good post describing this process for x86 and Cell architectures: Memory Address Translation.

Allocating pages

As an example, allocating just 10 bytes of memory using VirtualAlloc (the low-level allocation function on Windows) will allocate a whole page, that is 4096 bytes. You can access all of the 4096 bytes without triggering an access violation, eventhough you only requested 10 bytes.

Of course, page sizes differ across platforms (consoles). Some platforms even offer more than just one page-size. The reason for this is that because the TLB normally is of limited size, increasing the page-size can lead to fewer TLB misses (similar to cache misses), resulting in an increase in performance. However, larger pages can also lead to more wasted memory if you’re not careful. Thus, it’s a typical space/time-tradeoff.

Normally, the OS memory allocator (e.g. malloc/free) takes care of allocating pages, coalescing nearby allocations into contiguous regions, putting several small allocations on the same page, etc. However, as soon as we want to implement our own general-purpose allocator or any other custom allocation scheme, we need to be aware of such details, and cannot use malloc/free for our purposes.

Furthermore, knowing about such low-level details enables us to use a wealth of new debugging techniques like using protected pages, guard pages, etc. As an example, some pages could be marked read-only in order to find memory stomps, race conditions (writes on shared data), and more. Guard pages serve as a one-shot alarm for memory page access, and are e.g. used for growing an application’s stack. Applications like PageHeap use those features for finding memory accesses beyond an allocation’s boundary.

Use cases

MMU, TLB, pages, address translation, memory protection… that all sounds wonderful, but what can we do with it?

Because the OS clearly distinguishes between reserving address space (see MEM_RESERVE for the VirtualAlloc function) and allocating physical memory for address space (see MEM_COMMIT), we can build allocators that can grow to a specified upper limit, but only allocate the memory they actually need.

This is very, very useful when implementing growing allocators, because we can reserve a contiguous region of memory (=virtual address space), but only commit physical memory to it whenever we need it.

Virtual memory addressing is supported by all common desktop OSs (Windows, Linux, Mac), almost all consoles (cannot go into details because of NDAs) and even on mobiles like the iPhone. How different growing allocators can be built using virtual memory on those platforms will be the topic of the next posts!

Use cases

Pool allocators are extremely helpful for allocating/freeing objects of a certain size which often have to be created/destroyed dynamically, such as weapon bullets, entity instances, rigid bodies, etc.

Most of those objects are created/destroyed in completely random order, due to their dynamic nature. Therefore, it is desirable to be able to allocate/free memory with as little fragmentation as possible. Pool allocators are extremely well suited for that.

How does it work?

Simply put, a pool allocator allocates a chunk of memory once, and divides that memory into slots/bins/pools which fit exactly M instances of size N. As an example, consider we want to have a maximum of 256 bullets in flight at the same time, each bullet having a size of 32 bytes. Thus, the pool allocator would allocate 256*32 = 8192 bytes once, dividing it into slots which are then used for allocating/freeing objects of size 32.

But how are those allocations made? How can we guarantee O(1) time? How can allocations be made in any order, without fragmentation?

Freelists

The answer to all of the above boils down to the use of what is called a free list. Free lists internally store a linked-list of free slots inside the allocated memory. Storing them inplace is crucial – there is no std::vector, std::list, or similar that keeps track of free slots. It is all stored inside the pre-allocated pool of memory.

The way it is usually done is the following: each slot (32 bytes in our example) in the pool of memory is connected to the next slot simply by storing the pointer to the next slot in the first few bytes of the slot.

Assuming our pool of memory sits at location 0×0 in memory, the layout would be something like the following:

         +---------+---------+---------+---------+
         | 0x20    | 0x40    | 0x60    | nullptr |
         +---------+---------+---------+---------+

         ^         ^         ^         ^
         |         |         |         |
address: 0x0       0x20      0x40      0x60

The blocks denote the slots in memory, the bottom row shows the address in memory. As can be seen, the memory at 0×0 would contain a pointer to 0×20, which would contain a pointer to 0×40, and so on. We have just formed an intrusive linked-list in our memory pool.

There is one thing to note here: as long as slots are free, we can store anything we want inside those 32 bytes. When a slot is in use, we don’t need to store anything, because that slot is occupied anyway and so no longer part of our free list. All we have to do is remove a slot from the free list whenever it is allocated, and add it to the linked-list again whenever it is freed.

Implementation

Let us take a look at the inner workings of a free list:

class Freelist
{
public:
  Freelist(void* start, void* end, size_t elementSize, size_t alignment, size_t offset);

  inline void* Obtain(void);

  inline void Return(void* ptr);

private:
  Freelist* m_next;
};

The only member we need to store is a pointer to the free list, which simply acts as an alias in memory, and stores a pointer to a currently free slot in our memory pool.

Leaving alignment and offset requirements out of the equation for now, initializing a free list is quite simple:

union
{
  void* as_void;
  char* as_char;
  Freelist* as_self;
};

// assume as_self points to the first entry in the free list
m_next = as_self;
as_char += elementSize;

// initialize the free list - make every m_next of each element point to the next element in the list
Freelist* runner = m_next;
for (size_t i=1; i<numElements; ++i)
{
  runner->m_next = as_self;
  runner = as_self;
  as_char += elementSize;
}

runner->m_next = nullptr;

With the intrusive linked-list in place, allocating/freeing memory really becomes an O(1) operation, and is just ordinary linked-list manipulation code:

inline void* Freelist::Obtain(void)
{
  // is there an entry left?
  if (m_next == nullptr)
  {
    // we are out of entries
    return nullptr;
  }

  // obtain one element from the head of the free list
  Freelist* head = m_next;
  m_next = head->m_next;
  return head;
}

inline void Freelist::Return(void* ptr)
{
  // put the returned element at the head of the free list
  Freelist* head = static_cast<Freelist*>(ptr);
  head->m_next = m_next;
  m_next = head;
}

I moved the free list implementation to its own class so it can be used by both the non-growing and growing variant of the pool allocator. Furthermore, it is handy for other things, too.

Alignment requirements

In order to satisfy alignment requirements, we need to offset our slots into the memory pool once, so all slots can satisfy the same offset and alignment requirements. The disadvantage of this approach is that a pool allocator can never satisfy more than one offset requirement, but such cases should be very seldom.

The way it is done in Molecule is that users have to provide their maximum object size and maximum alignment requirement when constructing the pool allocator. The allocator can then satisfy all alignments <= maximumAlignment and object sizes <= maximumSize, and simply asserts in all other cases. This at least allows the user to allocate objects of different sizes (such as e.g. 4, 8, 12, 16) out of the same pool, with different alignments (such as e.g. 4, 8, 16, 32), if desired.

Control is in the user’s hands, so it is up to the user to either use different pools for different allocations (often recommended), or live with some wasted memory inside the memory pool.

Usage

Usage is simple. The following shows a free list which is able to satisfy allocations up to a size of 32 bytes and an alignment of 8 bytes. Note that the free list takes a range of memory in which it initializes itself. This ensures that free lists can be used for allocations on the stack, on the heap, and by different allocators (non-growing and growing).

ME_ALIGNED_SYMBOL(char memory[1024], 8) = {};

core::Freelist freelist(memory, memory+1024, 32, 8, 0);

// allocates a slot of 32 bytes, aligned to an 8-byte boundary
void* object0 = freelist.Obtain();

// allocates another slot of 32 bytes, aligned to an 8-byte boundary
void* object1 = freelist.Obtain();

// obtained slots can be returned in any order
freelist.Return(object1);
freelist.Return(object0);

The pool allocator described in this post simply has a Freelist instance as member, and forwards all Allocate() calls to freelist.Obtain(), and all Free() calls to freelist.Return(). Additionally, it asserts that allocation sizes and alignment requests fit the initial maximum sizes provided by the user.

Outlook

The pool allocator was the last remaining non-growing allocator we haven’t discussed yet. Starting with the next post, we will take a look at how to implement growing allocators by means of virtual memory and allocating physical pages from the OS.