Reference Handle Cap

[DoubleYou]

Below is a copied post by aers as relates to the reference handle cap. Posting it here, because having this information only on Reddit is not the best idea.

 

 

Quote

PSA: The reference handle cap, or diagnosing one of the causes of infinite load screens/CTDs on very heavy load orders

Note: This doesn't apply to every heavy load order, it's really dependent on having a high number of mods that make heavy world edits or add large numbers of cells that are not master files. Also, I'm pretty sure it applies to LE as well as SE, but I'm not checking all that myself.

This problem was brought to my attention thanks to Nukem's work on CK64, Ralgor actually encountering it in SSE, and some unrelated research of my own

There's a TL;DR at the bottom.

Persistent vs Temporary References

If you've ever browsed your plugins in xEdit you've probably noticed that cells have persistent and temporary objects placed in them. Arthmoor's got a write-up about this over here, but the important part is this:

A reference was either temporary, which means it was only loaded if it was within the grid surrounding the player, or it was persistent, which meant it was always loaded in memory at all times.

The way the engine handles this internally is pretty simple: all persistent references are loaded into memory before you even hit the main menu, and temporary references are loaded on-demand from your files as needed (for example, if a cell becomes active all its temporary references will load). The majority of these temporary references will then unload when they're no longer needed by the game.

Reference Handle Limit

Reference handles are how the rest of the engine looks up loaded reference objects (the technical details are pretty unimportant here; just know that every reference will have a handle). There's a cap of 2^20, or 1048576, active reference handles at any time. If you hit the cap, the game will either get stuck loading or just outright CTD or really do any number of things.

The Problem

2²⁰ is a pretty high limit and something you probably shouldn't hit during gameplay: there's only a limited number of cells actually loaded, and the vast majority of references are temporary, not persistent.

Until you actually look at the load process and realize that temporary reference loading/unloading is exclusive only to masters and not regular plugins. All temporary references from regular plugins are loaded into the game before the main menu. This includes if you overwrite a master's temporary reference in a plugin (every light source ELFX moves? yup, loaded at the start now).

To give some numbers that put this in perspective, Forgotten Dungeons SSE has 136356 temporary references. Vigilant has 110770. These two plugins alone will put you 1/4th of the way towards your game simply not working.

I want to clarify that this does not mean these are bad mods - Bruma has over 130k temporary references itself, for example - the problem here is that these mods are plugins when they probably should be masters. Presumably when this system was designed however many iterations of the engine pre-Skyrim ago the idea was that anything that added so many references was going to be a master.

Another thing to keep in mind is that during the course of gameplay a lot of references are created unique to your save game, so you can get to the point where your load order + your save game will start to break even if it seems like your base load order is fine.

Checking Your Load Order

Grab this xEdit script and put it in your *edit/Edit Scripts folder. Then load your entire load order into xEdit and right click -> Apply Script -> choose count_loaded_refs_in_load_order and click OK. This will take a few minutes to run, and then print out a log of all the refs in your load order that will load before you hit the main menu.

Found 0 temporary and 41433 persistent (41433 total) loaded references in [00] Skyrim.esm.

...

Found 67275 temporary and 125 persistent (67400 total) loaded references in [16] EnhancedLightsandFX.esp.

...

Found 112449 temporary and 4585 persistent (117034 total) loaded references in [67] Vigilant.esp.

...

Found 365172 temporary and 140045 persistent loaded references, for a grand total of 505217 loaded references.

The script isnt perfect but it will get a very close approximation (in this test load order I'm using, its ~2000 under what is actually reported in-game). If your total references is near 1048576, and honestly even if its above 900k~, you're in pretty dangerous territory. When Ralgor discovered his own load order was causing crashes due to hitting the reference handle cap, he was at ~1million and trying to enable DynDOLOD, which adds a bunch of references, was causing his game to break.

Solutions

One solution would be a plugin that raises the reference handle cap. CK fixes does this in the CK because its the only way to feasibly load a large worldspace plugin, since the CK has to load all temporary references as persistent anyway. Doing the same in SSE itself is a lot harder because of differences in compiler options.

Another potential solution is to hook the game engine to treat every plugin as a master for purposes of loading. I've been exploring this solution with the help of the xEdit team but there's nothing concrete decided yet.

The 'easy' solution then if you're at the limit or very close and its making your game unstable is to either remove the heaviest offenders from your load order or turn them into masters via xEdit. Keep in mind that making a plugin a master means it now has to load at the top of your load order with the other masters, before any plugins, which is why this might not be an easy solution depending on what the plugin in question is.

TL;DR

Temporary references in plugins don't behave the same way as temporary references in masters so you can easily hit the reference cap of 1048576 with a lot of plugins that add references. Use this xEdit script to scan your load order. If you're dangerously close to the cap remove the heaviest offenders or convert them into masters. Realistically you want to be nowhere near it because the longer you play the game the more references end up in your save game and the closer you could end up to hitting the limit.

Thanks to Ralgor for throwing together the xEdit script.

 

[z929669]

Thanks. I avoid Reddit, personally, since I hate the way conversations are (dis)organized and the interface.

[Mousetick]

Illustrated practical demonstration of the engine's handling of temporary references in ESM vs. ESP plugins

For mod authors and advanced mod users. TLDR: Skip to the Summary section further down.

• Part 1: The initial (vanilla) state

We're in Markarth, in front of the stables. Let's look at a few objects: the door, a handcart, and a hay bale:

Notice the console shows a Textures field for all the references. This indicates the textures are loaded and the object is rendered.

The door is a persistent reference because it's used with navmeshes for NPCs, among other things. The other objects are temporary references.

The door is only interesting as a point of temporary/persistent comparison later on.

• Part 2: The demonstration plugin

Now let's make a simple plugin for an hypothetical mod. We're going to permanently disable the handcart, make the hay bale bigger, and place a new small hay bale:

So the plugin overrides two vanilla references and adds a new reference. We've also given each reference an Editor ID to more easily look them up.

Let's go take a look at the changes in-game with our new plugin enabled. The handcart is disabled and no longer visible, the existing hay bale has grown in size, and there is a new small hay bale not far from it:

For the next tests, we'll be moving to Winterhold, far, far away at the opposite end of the map. In Winterhold, the cells we were in in Markarth (i.e. the active grid) won't be active any more.

• Part 3: Engine handling with an ESM plugin

The demonstration plugin is ESM-flagged. This is how Bethesda intended things to be based on how the engine is designed and behaves in Skyrim SE.

Let's look at our references from Winterhold. First the door. The door is still there, despite it being in Markarth and us in Wintherhold, as expected since it's a persistent reference. But note the Textures field is no longer present, meaning the door while still being loaded in the game, is no longer rendered. It all makes sense.

As for all the other references, they can't be found. They're no longer loaded in the game at all. It all makes sense again, since they're temporary in a cell outside Markarth and we're in Winterhold:

• Part 4: Engine handling with an ESP plugin

We remove the ESM flag from the demonstration plugin. This is how 99% of all mods made and released by mod authors are.

Let's again look at our references from Winterhold:

Surprise! They're all found. They're all loaded (but not rendered, as they have no Textures field). They're essentially treated like persistent references by the engine.

Note that even though the handcart is permanently disabled, it's still loaded. An object that will never ever seen in-game is wasting engine resources. Also note that the hay bale originally defined in Skyrim.esm, but has been overridden by our plugin, is now always loaded. The hay bale is now wasting engine resources.

We can even go into the Jar's Longhouse, an interior cell, and they're still there:

 

---

Summary

• The Skyrim SE engine is designed to handle (object or actor) references in ESM plugins, not in ESP plugins.
• References in ESP plugins still "work" but not as intended by Bethesda.
• Temporary references in ESP plugins essentially behave like persistent references: always loaded in the game, available and reachable globally from anywhere.
• All temporary references in ESP plugins, whether they are overriding existing temporary references, or newly added references, are affected.
• Temporary references in ESP plugins contribute to the total amount of loaded references, which has a hard limit.
• Even though temporary references in ESP plugins don't waste rendering resources, they still consume engine resources, mainly memory.

---

Miscellaneous Notes

• ESP plugins are bad not only because they're wasteful, but they also hide all the bugs in mods. Since all the references, temporary or persistent, are always loaded, there is no longer a meaningful distinction between persistent and temporary. A temporary reference may be used in the wrong context, it's all fine, it doesn't matter as it's always loaded, it always can be found. The same plugin would be a CTD crashfest if it were ESM, or some things wouldn't work correctly.
• All Creation Club modules, from the simplest gimmick to the largest content, are released by Bethesda as ESM plugins (.esl is the "light" version of .esm), for a simple reason: it's how plugins are expected to be in Skyrim SE.
• People can be confused by the term ESM which is commonly associated with game masters, i.e. the core game files (Skyrim.esm, Update.esm, etc.). And so they think it doesn't make sense for a mod or a patch to be ESM, to be a master. The two concepts must be disassociated. Game masters are the core game files, yes. But ESM is something else, it's just a small flag that has no relation with the nature or purpose of the plugin, but has big implications for the engine. Again, Creation Club modules are not core game files, they're Bethesda-approved mods, and yet they are all ESM.
• There is no downside to plugins being ESM, while ESP plugins exhibit proven disadvantages and deficiencies.
• Technically, ESM vs. ESP only matters for plugins that add new references or modify existing references. Those that don't can be ESP.

 

• Large references are not related to temporary or persistent references. They're a separate feature that is only available, and that is no coincidence, in ESM plugins. ESP plugins that override large references cause visual bugs, which is another reason ESP plugins are a bad idea.
• In our demonstration above, we only used static object references. The increased resource usage and waste caused by ESP plugins is worse with temporary NPC references (ACHR), because the engine must periodically evaluate and process their AI packages in case they might need to travel somewhere, or do something on a schedule. The engine is optimized to prioritize and refresh the most often the NPCs closest to the player, but it still must check all loaded NPCs from time to time.

 

• It's too late now to do anything about it. There's too much legacy with 99% of all existing mods using ESP plugins.
• Making existing ESP plugins into ESM can be trivial or extremely difficult.
• Even if some ESP plugins can be made ESM, there is the issue of sort orders and dependencies, particularly with automated sorting tools such as LOOT. This may require that other plugins be made ESM in order to satisfy the dependencies or sorting metarules.
• Nevertheless, certain kinds of mods which only deal with static (STAT/TREE/FLORA) or moveable static objects (MSTT) - such as tree mods; or landscape clutter mods - can and should be made ESM as they typically add or touch thousands of temporary references.

---

Persistent Location

There exists a sparely documented reference attribute named 'Persistent Location' which is extensively used by Bethesda in the vanilla plugins, but is apparently ignored or unknown by mod authors. It seems to be used exclusively with NPC references, so may only be intended to be used with them.

It allows a temporary ACHR to become "temporarily" persistent as long as the player is in the specified location. As a reminder, a location (LCTN record) is a "tag" that is applied to specific cells, which together belong to, and form, a geographic location.

The Persistent Location is an optimization feature: it avoids making an NPC reference persistent, which creates some overhead for the engine as discussed above, and instead keeps it as a temporary reference while allowing for some advanced NPC behavior across multiple cells, even across interior/exterior cells, while the player is in the geographic location.

For example, Adisla, a farmer working at the Hlaalu farm near Windhelm:

This minor NPC has advanced AI packages to make her sleep and eat inside, work and wander outside, on a specific schedule. She also has dialogue scenes with other NPCs on the farm. These features require her to be persistent. However her reference is temporary. But the Persistent Location attribute makes her persistent only when the player is in the Hlaalu Farm. Once the player exits the location the reference becomes temporary again, and completely unloads when the player moves further away.

Obviously this is only meaningful in ESM plugins. It's completely useless in ESP plugins, since all temporary references are always and everywhere pseudo-persistent.

[MisterMorden]

Wow, awesome post, MT!  Information overload, lol.

[z929669]

On 9/13/2023 at 2:17 PM, Mousetick said:

Illustrated practical demonstration of the engine's handling of temporary references in ESM vs. ESP plugins

For mod authors and advanced mod users. TLDR: Skip to the Summary section further down.

• Part 1: The initial (vanilla) state

We're in Markarth, in front of the stables. Let's look at a few objects: the door, a handcart, and a hay bale:

Notice the console shows a Textures field for all the references. This indicates the textures are loaded and the object is rendered.

The door is a persistent reference because it's used with navmeshes for NPCs, among other things. The other objects are temporary references.

The door is only interesting as a point of temporary/persistent comparison later on.

• Part 2: The demonstration plugin

Now let's make a simple plugin for an hypothetical mod. We're going to permanently disable the handcart, make the hay bale bigger, and place a new small hay bale:

So the plugin overrides two vanilla references and adds a new reference. We've also given each reference an Editor ID to more easily look them up.

Let's go take a look at the changes in-game with our new plugin enabled. The handcart is disabled and no longer visible, the existing hay bale has grown in size, and there is a new small hay bale not far from it:

For the next tests, we'll be moving to Winterhold, far, far away at the opposite end of the map. In Winterhold, the cells we were in in Markarth (i.e. the active grid) won't be active any more.

• Part 3: Engine handling with an ESM plugin

The demonstration plugin is ESM-flagged. This is how Bethesda intended things to be based on how the engine is designed and behaves in Skyrim SE.

Let's look at our references from Winterhold. First the door. The door is still there, despite it being in Markarth and us in Wintherhold, as expected since it's a persistent reference. But note the Textures field is no longer present, meaning the door while still being loaded in the game, is no longer rendered. It all makes sense.

As for all the other references, they can't be found. They're no longer loaded in the game at all. It all makes sense again, since they're temporary in a cell outside Markarth and we're in Winterhold:

• Part 4: Engine handling with an ESP plugin

We remove the ESM flag from the demonstration plugin. This is how 99% of all mods made and released by mod authors are.

Let's again look at our references from Winterhold:

Surprise! They're all found. They're all loaded (but not rendered, as they have no Textures field). They're essentially treated like persistent references by the engine.

Note that even though the handcart is permanently disabled, it's still loaded. An object that will never ever seen in-game is wasting engine resources. Also note that the hay bale originally defined in Skyrim.esm, but has been overridden by our plugin, is now always loaded. The hay bale is now wasting engine resources.

We can even go into the Jar's Longhouse, an interior cell, and they're still there:

 

---

Summary

• The Skyrim SE engine is designed to handle (object or actor) references in ESM plugins, not in ESP plugins.
• References in ESP plugins still "work" but not as intended by Bethesda.
• Temporary references in ESP plugins essentially behave like persistent references: always loaded in the game, available and reachable globally from anywhere.
• All temporary references in ESP plugins, whether they are overriding existing temporary references, or newly added references, are affected.
• Temporary references in ESP plugins contribute to the total amount of loaded references, which has a hard limit.
• Even though temporary references in ESP plugins don't waste rendering resources, they still consume engine resources, mainly memory.

---

Miscellaneous Notes

• ESP plugins are bad not only because they're wasteful, but they also hide all the bugs in mods. Since all the references, temporary or persistent, are always loaded, there is no longer a meaningful distinction between persistent and temporary. A temporary reference may be used in the wrong context, it's all fine, it doesn't matter as it's always loaded, it always can be found. The same plugin would be a CTD crashfest if it were ESM, or some things wouldn't work correctly.
• All Creation Club modules, from the simplest gimmick to the largest content, are released by Bethesda as ESM plugins (.esl is the "light" version of .esm), for a simple reason: it's how plugins are expected to be in Skyrim SE.
• People can be confused by the term ESM which is commonly associated with game masters, i.e. the core game files (Skyrim.esm, Update.esm, etc.). And so they think it doesn't make sense for a mod or a patch to be ESM, to be a master. The two concepts must be disassociated. Game masters are the core game files, yes. But ESM is something else, it's just a small flag that has no relation with the nature or purpose of the plugin, but has big implications for the engine. Again, Creation Club modules are not core game files, they're Bethesda-approved mods, and yet they are all ESM.
• There is no downside to plugins being ESM, while ESP plugins exhibit proven disadvantages and deficiencies.
• Technically, ESM vs. ESP only matters for plugins that add new references or modify existing references. Those that don't can be ESP.

 

• Large references are not related to temporary or persistent references. They're a separate feature that is only available, and that is no coincidence, in ESM plugins. ESP plugins that override large references cause visual bugs, which is another reason ESP plugins are a bad idea.
• In our demonstration above, we only used static object references. The increased resource usage and waste caused by ESP plugins is worse with temporary NPC references (ACHR), because the engine must periodically evaluate and process their AI packages in case they might need to travel somewhere, or do something on a schedule. The engine is optimized to prioritize and refresh the most often the NPCs closest to the player, but it still must check all loaded NPCs from time to time.

 

• It's too late now to do anything about it. There's too much legacy with 99% of all existing mods using ESP plugins.
• Making existing ESP plugins into ESM can be trivial or extremely difficult.
• Even if some ESP plugins can be made ESM, there is the issue of sort orders and dependencies, particularly with automated sorting tools such as LOOT. This may require that other plugins be made ESM in order to satisfy the dependencies or sorting metarules.
• Nevertheless, certain kinds of mods which only deal with static (STAT/TREE/FLORA) or moveable static objects (MSTT) - such as tree mods; or landscape clutter mods - can and should be made ESM as they typically add or touch thousands of temporary references.

---

Persistent Location

There exists a sparely documented reference attribute named 'Persistent Location' which is extensively used by Bethesda in the vanilla plugins, but is apparently ignored or unknown by mod authors. It seems to be used exclusively with NPC references, so may only be intended to be used with them.

It allows a temporary ACHR to become "temporarily" persistent as long as the player is in the specified location. As a reminder, a location (LCTN record) is a "tag" that is applied to specific cells, which together belong to, and form, a geographic location.

The Persistent Location is an optimization feature: it avoids making an NPC reference persistent, which creates some overhead for the engine as discussed above, and instead keeps it as a temporary reference while allowing for some advanced NPC behavior across multiple cells, even across interior/exterior cells, while the player is in the geographic location.

For example, Adisla, a farmer working at the Hlaalu farm near Windhelm:

This minor NPC has advanced AI packages to make her sleep and eat inside, work and wander outside, on a specific schedule. She also has dialogue scenes with other NPCs on the farm. These features require her to be persistent. However her reference is temporary. But the Persistent Location attribute makes her persistent only when the player is in the Hlaalu Farm. Once the player exits the location the reference becomes temporary again, and completely unloads when the player moves further away.

Obviously this is only meaningful in ESM plugins. It's completely useless in ESP plugins, since all temporary references are always and everywhere pseudo-persistent.

I think ESP can be though of as "Elder Scrolls Plugin Persistent". I think the ESP was created for simplicity in testing specific things during development and, early on, exposed (or interpreted) as the simplest way to prevent failure from user (modder) error. Like dumbing things down a bit as an insurance policy of sorts. Why else would this game-file type have ever been exposed in the first place, given the seemingly plain & simple consequences (as you illustrated nicely in your examle)?

For me, Starfield has underscored the relevance of this topic.

Edit: ... OP notwithstanding.

[Mousetick]

1 hour ago, z929669 said:

Why else would this game-file type have ever been exposed in the first place, given the seemingly plain & simple consequences (as you illustrated nicely in your examle)?

Good question. I don't know the history of past BGS game releases, their official modding tools and associated documentation, that could shed some light.

ESP might be a legacy leftover from previous games and tools that is still supported by the current games.

Or it might have been intended by Bethesda for temporary quick-and-dirty patches (ESP as in Patch) which naturally and conveniently sort after the ESMs, in between official game releases or during development.

From my uninformed outsider's point of view, looking at the pattern of past games up until the latest Starfield, Bethesda don't design their games with moddability in mind. Moddability is bolted on organically later as an afterthought. The people who build the engine may not be the same as the people who build the CK, there may be technical gaps in between. Furthermore, Bethesda doesn't provide any official public documentation or communication on anything for modders to rely on.

What modders have done with the games and the tools happened largely out of Bethesda's control or help. A lot of modding was likely unintended or unenvisioned by Bethesda. All that may change with the new BGS Creations platform.

I think trying to explain the current state of affairs in regards to ESP vs. ESM and how we got here is a little futile, because there is probably no rational explanation. It just happened without Bethesda willing it, and they didn't care enough to explain and correct it. Though they did care about the CC content which is ESM. 

Edited December 11, 2023 by Mousetick

[z929669]

Yeah, it's futile but interesting. It's simple enough information to share, given BSG want's to support the modding community, and the community itself has gleaned plenty on their own. Why they haven't surfaced certain relevant info related to modding recommendations all these years is perplexing to me. We can speculate that it was an oversight, but that seems unlikely to me.

Ultimately, it doesn't matter though. The point is that converting relevant ESP into ESM seems like it should be a 'thing', but MAs probably just don't want to deal with the support headaches about why their mods "aren't working". It may be progressive to provide optional ESM variants to certain mods, stating the reasoning.

[Mousetick]

1 hour ago, z929669 said:

The point is that converting relevant ESP into ESM seems like it should be a 'thing', but MAs probably just don't want to deal with the support headaches about why their mods "aren't working".

Another obstacle to converting existing ESP plugins to ESM is that it changes their sort order relative to other plugins. MAs' recommendations to manually sort their mod before or after some other mod become void. LOOT's meta rules for automatic sorting become bogus.

The recent xEdit update (4.1.5) doesn't allow creating or saving Starfield plugins unless they're ESM. That should help ensure all Starfield mods are ESM from the get go. Assuming all MAs use xEdit, but many don't. What the successor to CK will do and allow for Starfield, or not, remains to be seen.