In my last post I finished by showing how I loaded obj models directly into the engine. I also complained that it was taking a horrendously long time to load (especially for Debug builds). I looked around for faster ways to load obj's, but there really weren't any... (sort of*) Why aren't there any obj loader libraries?
*There is assimp, but I'll get to that further down
One answer would be that OBJs weren't designed for run-time model loading. Computers don't like parsing text. They would rather read binary; things have set sizes and can be read in chunks rather than single characters at a time. So next I looked around for a binary file format that would be faster to load. "Why re-invent the wheel" I thought?
The problem is that standardized run-time binary file formats don't really exist either. This when it really dawned on me. For run-time, there's no point in storing things your engine doesn't need. And more than that, it would be great if the data you store is in the correct format for your engine. For example, you could store the raw vertex buffer data so you can directly cast it into DirectX vertex buffer data. Obviously, it would be extremely hard to get people to agree upon a set standard of what is "necessary", so it's common practice to have a specific binary file format for the engine that is specifically tailored to make loading the data as fast and easy as possible.
Therefore, I set out to make my own binary file format, which, to stay with the Halfling theme, I dubbed the 'Halfling Model File'. Every indent represents a member variable of the level above it. 'String data' and 'Subset data' are arrays. (The format of the blog template makes the following table a bit hard to read. There is an ASCII version of the table here if that's easier to read)
Item | Type | Required | Description |
File Id | '\0FMH' | T | Little-endian "HMF\0" |
File format version | byte | T | Version of the HMF format that this file uses |
Flags | uint64 | T | Bitwise-OR of flags used in the file. See the flags below |
String Table | F | ||
Num strings | uint32 | T | The number of strings in the table |
String data | T | ||
String length | uint16 | T | Length of the string |
String | char[] | T | The string characters. DOES NOT HAVE A NULL TERMINATION |
Num Vertices | uint32 | T | The number of vertices in the file |
Num Indices | uint32 | T | The number of indices in the file |
NumVertexElements | uint16 | T | The number of elements in the vertex description |
Vertex Buffer Desc | D3D11_BUFFER_DESC | T | A hard cast of the vertex buffer description |
Index Buffer Desc | D3D11_BUFFER_DESC | T | A hard cast of the index buffer description |
Instance Buffer Desc | D3D11_BUFFER_DESC | F | A hard cast of the instance buffer description |
Vertex data | void[] | T | Will be read in a single block using VertexBufferDesc.ByteWidth |
Index data | void[] | T | Will be read in a single block using IndexBufferDesc.ByteWidth |
Instance buffer data | void[] | F | Will be read in a single block using InstanceBufferDesc.ByteWidth |
Num Subsets | uint32 | T | The number of subsets in the file |
Subset data | Subset[] | T | Will read in a single block to a Subset[] |
Vertex Start | uint64 | T | The index to the first vertex used by the subset |
Vertex Count | uint64 | T | The number of vertices used by the subset (All used vertices must be in the range VertexStart + VertexCount) |
Index Start | uint64 | T | The index to the first index used by the subset |
Index Count | uint64 | T | The number of indices used by the subset (All used indices must be in the range IndexStart + IndexCount) |
Material Ambient Color | float[3] | T | The RGB ambient color values of the material |
Material Specular Intensity | float | T | The Specular Intensity |
Material Diffuse Color | float[4] | T | The RGBA diffuse color values of the material |
Material Specular Color | float[3] | T | The RGB specular color values of the material |
Material Specular Power | float | T | The Specular Power |
Diffuse Color Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. |
Specular Color Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. |
Specular Power Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. |
Alpha Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. |
Bump Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. Mutually exclusive with Normal Map |
Normal Map Filename | int32 | T | An index to the string table. -1 if it doesn't exist. Mutually exclusive with Bump Map |
I designed the file format to make it as easy as possible to cast large chunks of memory directly from hard disk to arrays or usable engine structures. For example, the subset data is read in one giant chunk and cast directly to an array.
There's only one problem: Binary is not really human-readable. It would be extremely arduous to create a HMF file manually, so I created a tool to automate the task. While my hand-written obj-parser fulfilled its purpose, it's was pretty bare-bones and made quite a few assumptions. Rather than spend the time to beef it up to what was necessary, I leveraged the wonderful tool ASSIMP. ASSIMP is a C++ library for loading arbitrary model file formats into a standard internal representation. It also has a number of algorithms for optimizing the model data. For example, calculating normals, triangulating meshes, or removing duplicate vertices. Therefore, I use ASSIMP to load and optimize the model, then I output ASSIMP's mesh data to the HMF format. The source code is a bit too long to directly post here, so instead I'll link you to it on GitHub. I'll also point you to a pre-compiled binary of the tool here.
As I was writing the the code for the tool, it became apparent that I needed a way for the user to tell the tool certain parameters about the mode. For example, what textures do you want to use? I could have passed these in with command line arguments, but that's not very readable. Therefore, I put all the possible arguments into an ini file and then have the user pass the path to the ini file in as a command line arg. Below is the ini file for the sponza.obj model:
[Post-Processing] ; If normals already exist, setting GenNormals to true will do nothing GenNormals = true ; If tangents already exist, setting GenNormals to true will do nothing CalcTangents = true ; The booleans represent a high level override for these material properties. ; If the boolean is false, the property will be set to NULL, even if the property ; exists within the input model file ; If the boolean is true, but the value doesn't exist within the input model file, ; the property will be set to NULL [MaterialPropertyOverrides] AmbientColor = true DiffuseColor = true SpecColor = true Opacity = true SpecPower = true SpecIntensity = true ; The booleans represent a high level override for these textures. ; If the boolean is false, the texture will be excluded, even if the texture ; exists within the input model file ; If the boolean is true, but the texture doesn't exist within the input model ; file properties, the texture will still be excluded [TextureOverrides] DiffuseColorMap = true NormalMap = true DisplacementMap = true AlphaMap = true SpecColorMap = true SpecPowerMap = true ; Usages can be 'default', 'immutable', 'dynamic', or 'staging' ; In the case of a mis-spelling, immutable is assumed [BufferDesc] VertexBufferUsage = immutable IndexBufferUsage = immutable ; TextureMapRedirects allow you to interpret certain textures as other kinds ; For example, OBJ doesn't directly support normal maps. Often, you will then see ; the normal map in the height (bump) map slot. These options allow you to specify ; what texture goes where. ; ; Any Maps that are excluded are treated as mapping to their own kind ; IE. excluding DiffuseColorMap is interpreted as: ; DiffuseColorMap = diffuse ; ; The available kinds are: 'diffuse', 'normal', 'height', 'displacement', 'alpha', ; 'specColor', and 'specPower' [TextureMapRedirects] DiffuseColorMap = diffuse NormalMap = height DisplacementMap = displacement AlphaMap = alpha SpecColorMap = specColor SpecPowerMap = specPower
So with that we now have a fully functioning binary file format! And more than that, with a few changes in the engine code, we can load the scene cold in less than 2 seconds! (It's almost instant if your file cache is still hot). (Pre-compiled binaries here).
Well that's it for now. As always, feel free to ask questions and comment.
Happy coding
-RichieSams
No comments:
Post a Comment