Havok middleware

Havok has been around since 2000s. It is commonly associated with game physics, however there is more to it than meets the eye.
Havok has and is still used in countless games to this day. It is a almost complete toolset for numerous problematics.

This topic will cover broad solutions so reader experience will be required about:

• Basic knowledge about binary formats
• Basic knowledge about XML format
• How ELF or PE works
• C/C++ data structure layout, member alignment, vtables
• Data endianess
• Virtual memory

So, what Havok covers?

• Animation (presumably discontinued), covers animation algorithms, animation compression and their bindings. It is used to animate transforms, parameters and triggers (aka annotations).
  • Animation classes have hka prefix.
• Behavior (presumably discontinued), is used for blending animations, inverse kinematics and overall animation control. It is a set of state machines and is very similar to Animation Blueprints used in Unreal Engine 4.
  • Behavior classes have hkb prefix.
• AI is used to create and control artificial intelligence (the old one) notably navigation (via nav meshes) and path finding. It can be linked to physics world as it can be used for navigation.
  • AI classes have hkai prefix.
• Destruction is used to generate and work with dynamic destruction and object fracture and deformations. It is similar to Chaos in Unreal Engine 4.
  • Destruction classes have hkd prefix for legacy and hknd prefix for new versions.
• Physics is probably most used and most known usage. It’s used to define entire world of physics, it’s shapes, constraints and algorithms for collisions and physics that ties everything together.
  • Physics classes have hkp prefix for legacy and hknp, hkcd prefix for new versions.
• Extra stuff, like models for debug rendering, scripting engine and metadata are also present.
  • Extra classes have hkx prefix.

In addition to these above solutions, developers can integrate their custom classes into Havok itself.

Why care?

Havok uses numerous file formats that can store any data and it’s entirely up to the developers what format they want use or even if they want to use them at all.
Havok uses at least 5 file formats.

Packfile format

NOTE: In order to fully understand packfile format, basic knowledge about C/C++ language is essential. This format is universal store format that is almost always used.

It’s mostly used in a binary form or very rarely in XML form
Packfiles are using sections in order to separate different kind of data.
Most common sections are:

• __classnames__ that store all classes used within packfile, it stores only class name and it’s signature hash. It’s always used in binary format, never in XML.
• __types__ stores complete class information, their names, base classes, member names and member types. This section is optional and is mostly omitted from packfiles.
• __data__ finally store data values

XML Packfile

Stores data as XML text. This format is platform independent but slow to load compared to it’s binary counterpart.
Following code snipped describes XML layout for packfiles.

<hkpackfile>
    <hksection name="__types__">
    <hkobject class="hkClass" name="hkGroupFilterDesc">
        <hkparam name="name">hkGroupFilter</hkparam>
        <hkparam name="declaredMembers" numelements="1">
        <struct>
            <hkparam name="name">nextFreeSystemGroup</hkparam>
        </struct>
        </hkparam>
    </hkobject>
    </hksection>
</hkpackfile>

Packfile always starts with `hkpackfile` node. This node may have additional attributes like packfile version, SDK version, or other metadata.
Only children allowed for `hkpackfile` node are `hksection` with designated name attribute describing section.
Only children allowed for `hksection` node are `hkobject` that will encapsulate data for specified class. When XML is loaded, it will create class instance. Class name attribute is mostly used for referencing so other classes can use it.
Every `hkobject` must have `hkparam` children. Eventually they store all the data as text format. Such data is further parsed and then assigned to designated class member.

For example, above XML packfile will represent data for following class declaration in C++class

struct hkClassMember {
  hkString name;
};

class hkClass {
  hkString name;
  hkArray<hkClassMember> declaredMembers;
};

Binary packfile

This format is a first choice to store data. It’s super fast to load and data are in native format. That’s why it’s first choice and is commonly used.
Under the hood, this format is same as ELF or PE (executables or objects files produced by compilers), because it stores data as memory images.
Packfiles are loaded into virtual memory as a whole and then patched (patching occurs for pointers and class vtables). In one way, they are an extension to the main executable.

NOTE:. If you want to truly understand the complexity of such format, do research on ELF format.

This format have certain limitations, it’s platform specific. And I will shortly explain why.
But firstly let’s explore file header.

struct hkxHeaderData {
  u32 magic1; // 0x57E0E057
  u32 magic2; // 0x10C0C010
  u32 userTag;
  u32 version;
  u8 pointerSize;
  bool littleEndian;
  bool reuseBaseClassPadding;
  bool emptyBaseClassOptimization;
  s32 numSections;
  s32 contentsSectionIndex;
  s32 contentsSectionOffset;
  s32 contentsClassNameSectionIndex;
  s32 contentsClassNameSectionOffset;
  char contentsVersion[16];
};

Where:

• magic1 and magic2 are two endian independent magic numbers.
• userTag can be any number specified by user, it’s metadata and does not affect packfile in any way.
• version is a version of packfile, this does not have affect on data in any way except layout of section and header data.
• pointerSize can be either 4 for 32bit systems or 8 for 64bit systems.
• littleEndian will specify data endianess. If false, big endian is used.
• reuseBaseClassPadding is compiler specific, it is used mostly by GNU GCC compiler. In order to understand do research on data layouts and struct data padding for in C language.

Some examples of common data layouts:

Platform	pointerSize	littleEndian	reuseBaseClassPadding
Windows x64	8	✔	❌
Windows x86	4	✔	❌
XBOX 360	4	❌	❌
PS3	4	❌	✔
Linux x64 or PS4	8	✔	✔

And here comes the biggest limitation of this format. You cannot load packfile from different platform.
For example packfile from PS3 cannot be possibly loaded on Windows x64 platform, because the data layout is different than platform it’s being loaded into.

This is it for now. I can provide more info if there are requests for it.

Tagfile

I won’t go into depth with this format, since it’s very rarely used.
Tagfile are platform independent. They can be as XML or as binary data. XML schema is different than packfile. It must have class type info and in addition does not store default data.
Binary format starts with 0xDO11FACECAB00D1E magic number.
The rest of data is very similar to Profobuf Wire format or JSON binary formats like CBOR, BSON, MsgPack or others.
Again, I won’t go into detail unless there is an interest in it.

New format

This format is used since 2015 and replaces packfiles. In it’s own sense, it’s hybrid format from both packfiles and tagfiles.
Tagfiles are no longer loaded as a whole and data is stored into data chunks instead of sections.

Each data chunk begins with following structure:

struct hkChunk {
    u8 flags;
    u24 size;
    u32 chunkMagic;
};

Where:

• flags describe chunk, there seems to be only 0x40 which indicates either following chunk is not group
• size tells size of entire chunk including hkChunk, data are in big endian.
• chunkMagic is chunk type and is in ASCII

Some chunk types:

• TAG0: Root chunk where everything begins
• SDKV: String value containing SDK version (and possibly layout data)
• DATA: Contains native class data just like in packfiles
• TYPE: Marks groups with class and type descriptions
• TSTR: Type names
• TNAM: Index that is used to build type names
• FSTR: Class member names
• TBOD: Index that is used to build classes or types
• THSH: Class signatures
• INDX: Marks groups with pointer patches for DATA
• ITEM: Index for class entries
• PTCH: Index for pointer patches

Chunks are grouped as:

TAG0 {
    SDKV,
    DATA,
    TYPE {
        TPTR,
        TSTR,
        TNAM,
        FSTR,
        TBOD,
        THSH,
        TPAD,
    },
    INDX {
        ITEM,
        PTCH,
    }
}

Known tools

Project Anarchy

Project Anarchy was official Havok SDK for registered developers durin Intel acquisition era.
It contained Preview Tool to preview 3D shapes and simulate physics.
Content Tools was set of build tools to generate and compile Havok data to numerous formats for all platforms.
And Havok source code with compiled libraries for legacy physics and animations.
It also contained official plugins for Autodesk software like 3ds max or Maya.
Official website is no longer available, but there are backups online. Project ararchy contained versions from 550 to 2014.

HavokLib

Unofficial open source library currenty only for Havok Animation. This library is focused on multi-platform binary packfile loading. It can convert binary packfiles (old and new) to XML. HavokLib also contains cli toolset for conversion of Havok animation data into GLTF format.

HavokLib can be found at: https://github.com/PredatorCZ/HavokLib

HavokMax

Unofficial native plugin for 3ds max that uses HavokLib. Can import binary packfiles (old and new). Can export skeletal and animation data to XML format. HavokMax is no longer maintained and is only as archive.

HavokMax can be found at: https://github.com/PredatorCZ/HavokMax

Detailed info can be found at: https://lukascone.wordpress.com/2019/03/21/havok-3ds-max-plugin

HavokNoesis

Unofficial native plugin for Noesis that uses HavokLib. Can import binary packfiles (old and new). Can export skeletal and animation data to XML format. HavokNoesis is very old, it’s possible to update it, if there is legitimate interest.

HavokNoesis can be found at: https://github.com/PredatorCZ/HavokNoesis

Detailed info can be found at: https://lukascone.wordpress.com/2019/05/13/havok-noesis-plugin

This is it for now

I haven’t even scraped the surface of this behemoth of a middleware.
It took way too much time than I anticipated to make even this little text.
As I mentioned mutiple times, more research can be added. If there is a demand.
Feel free to share your research as well.
Thank you for reading this.

NOTE:
This page is an archive of ResHax: Havok middleware