One of the features of Falcon, Duality’s digital twin simulator, is the ability to load assets such as 3D models, audio, textures, python scripts, etc. at runtime without the need to repackage/cook the project as normally happens with Unreal Engine game distributions.
It may sound hard to believe, but it is definitely not a trivial task: most of the Unreal optimizations are possible because assets are pre-cooked and optimized in the editor, and a lot of those optimizations/analysis features are not available at runtime, for example: automatic LODs generation API is not exposed in a packaged build (as well as the distance fields generation required by Nanite in UE5).
While Falcon can load different types of assets, I will focus on 3D mesh loading, because it was one of the major bottlenecks that the Duality team had to deal with.
As of April 2022, Falcon was able to load meshes at runtime from USD assets (Pixar format), glTF (Khronos standard) and from Unreal Engine pak files. The last one may surprise UE developers who may not be aware that with a bit of C++, cooked and packaged assets can be loaded in running builds of their projects.
So, we have three different kinds of assets, and potentially all of them can bring the same result: why would someone choose one or the other?
Let’s start with a bit of context related to Falcon and I will use a metaphor that Duality’s Founder & CEO, Apurva Shah, likes to use: Falcon’s tech stack for digital twin simulation resembles the web: Unreal Engine is the browser, USD is the HTML and Python is the Javascript.
So USD is a core part of how Falcon users define a Simulation Scenario.
“But wait, isn’t USD a format for defining meshes (and other media)? ”
Yep, sure, but it is a strongly structured format with concepts like vectors and matrices, properties, scopes…
Compared to a plain JSON file USD offers a “type-safer” approach in describing complex, spatial hierarchies and associated metadata. The fact that most of the types are used for defining geometries is just an advantage for Falcon: it means that in the same file you can have both Scenarios definitions (which digital twins you need to place, and where the geospatial information about the environment, robots properties, etc.) as well as geometric informations for both visual and physical properties.
If USD is able to describe both the simulation scenario as well as associate geometry, why support glTF and Unreal pakfiles as well?
The reason is simple: I omitted a crucial detail, USD parsing, compared to glTF or plain pakfiles loading, is complex and slow, really slow… – and there is nothing wrong with that given the type safety and advanced features to combine and override attributes such as class inheritance, variants, etc. – blaming a USD parser for being slow would be similar to blaming a C++ compiler for being slower than a bytecode interpreter.
“Slow” is a scary word in computer science, and like any human word, it means nothing for a machine if we do not convert it in a numeric form. So let’s see what slow means here.
I am going to compare the ubiquitous Sponza scene and its loading into Falcon using USD, glTF and as a pakfile.
But before reviewing the numbers, let’s consider exactly how those assets are loaded:
USD: We use the official Pixar libraries, both from C++ and Python (the Python modules are just wrappers over the C++ libraries, in fact performance differences are quite neglectable). The Sponza original glTF asset has been converted to USD (.usda and .usdc) using Blender (and by separating the single mesh by materials given that the Unreal USD importer does not support geometry subsets).
glTF: We use the glTFRuntime open source plugin (https://github.com/rdeioris/glTFRuntime pure C++, Note: I am the author of it, but I swear I did not cheat with the numbers 😛) Additional Note: glTFRuntime has way more features than the USD runtime parser, so take this into account too)
Pakfiles: We have imported the Sponza scenario in Unreal 5.0.1 (using glTFRuntime packaging feature to grant the same geometric structure) and cooked/packaged it in a pakfile.
An additional note, for Falcon users working with on-prem systems, we load the assets from the local filesystem and not from external storages like S3 or plain HTTP servers.
Here are the results from 3 different executions of Falcon running on a Threadripper 3960. We account only for the loading time (ensuring that the Unreal assets are not preloaded before loading the scenario). Rendering time is not relevant here.
Okay, we can clearly see how loading a USD scene is slower (but not too much in the case of the binary variant, and this is probably the most interesting set of numbers in the table) in comparison to loading from a glTF file and definitely (about 6 times) slower than a Pakfile. How is this possible? Are we not talking about the same amount of vertices?
Pakfiles (well i should say UAssets, pakfiles are archives of them) have their data serialized in a way that is handy for being uploaded to the GPU but generating them requires the user to have Unreal installed, importing assets from DCC tools and running a command line script for triggering the cooking of the assets and the pakfile generation. The process is slow and painful – especially because you need to take into account all of the dependencies of your scene, including material functions. But it tries to optimize your assets too, so it is definitely worth it.
Another interesting result is that once the asset is loaded the first time, the following attempts to load require less than 30 milliseconds for a GPU upload, while the glTF (that has its internal caching system for avoiding reparsing) requires about double. This shows how powerful the Epic optimizations are when processing/cooking the assets.
glTF also benefits from IO and GPU friendly serialization, but the implementation has to parse a pretty complex data structure and eventually convert non-compatible sets of information and that has associated costs. If you are familiar with glTF you may ask yourself if using GLB (the binary variant of glTF) makes any difference. Long story short: No! GLB files are only combinations of binary data + the json text blob in a single file. The (slow) json parsing logic is still here.
USD instead, has been engineered from the ground up to be an ‘exchange’ format between CG artists. It is very strict (and this is a good thing for visualization formats). Pixar itself offers the SDK source for both creating and parsing assets under a permissive license. This format allows the artist to export basically everything created without compromises. Definitely a very valuable feature during collaborative authoring of assets.
On the other side, glTF is meant to be efficient, especially for the web. It includes only the minimal amount of information for loading the data and parsers can implement ‘extensions’ that are generally standardized by Khronos itself. There are dozens of extensions, but expecting your parser or your DCC to support all of them is fairly tricky, and I have to admit that whenever I add support for a new extension in glTFRuntime I am really scared about impacting performance…
Let’s be honest, the main advantage of USD (from a content perspective) is that Pixar offers a high quality SDK for it. This solves a lot of issues (basically all of the implementations, including Unreal and Blender use it, and I see no reasons for not doing it), but again to support all of those features the SDK has to be complex – and complexity has costs.
Back to Falcon and Duality customers, while we suggest using USD when exchanging assets, the reality is that the Epic USD importer is still a work in progress. (We have been positively impressed by the Blender 3.1.2 one!) But this means that when you import a USD asset to Unreal (for converting it as a UAsset) you may end up with weird stuff. (Remember, we needed to split the mesh by material in Blender to overcome Unreal limits, and this is just a super-simple case). And the same is true for the experimental Epic glTF importer.
So USD (still) is not the way to go for exchanging assets when working with Unreal (that is heavily FBX centric) but it is definitely the format to go (for us) as a “Scenario description format” given its rich and solid support for geometric and visual friendly types.
This is an example of a simple scenario based on an Unreal engine level with a character loaded in a glTF scenario (Sponza!) with a bunch of active sensors attached to its ‘pelvis’ bone and a python script:
#usda 1.0
(
upAxis = "Z"
)
def Scope "Simulation"
{
def Scope "World"
{
def ULevel "HomeLevel"
(
assetInfo = {
string asset_path = "/Game/SimpleLevel"
}
)
{}
def UAsset "Sponza" (
assetInfo = {
asset asset_path = @glTF/Sponza.gltf@
}
)
{
string CollisionMode = "Complex"
float3 xformOp:scale = (3, 3, 3)
uniform token[] xformOpOrder = ["xformOp:scale"]
}
}
def Scope "Characters"
{
def Character "Manny"
(
assetInfo = {
string asset_path = "/Game/ThirdPersonBP/Blueprints/ThirdPersonCharacter.ThirdPersonCharacter_C"
}
)
{
def Socket "pelvis"
{
def DepthSensor "Depth"
{
string Format = "U16"
float DepthSensorRange = 10
}
def CameraSensor "Camera"
{
string Format = "BGRA"
bool bHDR = true
}
}
string PythonModule = "robot_logic"
string PythonClass = "AutoMove"
}
}
}
Even if you have never used Falcon, the Scenario description should be pretty self-explanatory and it;s important that we can augment the CG-related features of the USD format with other approaches when performance is a bottleneck, for example by directly referencing a Unreal or a glTF assets (and potentially others formats in the future…).
