Large Reference Bug Fix Explained

[z929669]

UPDATE: "Best answer" has sheson's succinct details and 'rules' for LR avoidance/fix as well as responses corresponding to this OP, but do see Mousetick's posts for a deeper dive into plugin-patching specifics relevant to resolving the LR bug.

---

@sheson & @Mousetick have some very informative exchanges about the large references (LR) issue and it's mitigation, so that's why you are both tagged here. (search "large reference" and "large references" on the DynDOLOD alpha topic to find these posts). Example (follow the quote links to go up the convo chain)

After spending hours reading through the DynDOLOD doc on this and about twenty relevant posts in the DynDOLOD topic, I still don't think the large reference mitigation methods are clear. I'm creating this topic for posterity and as a placeholder for anyone to understand methods to fix and their limitations.

The issue itself and why it happens is clear in the DynDOLOD documentation. It's the mitigations/fixes and WHY they work that's unclear and even a bit contradictory, in some cases:

1. The seeming contradiction I mentioned: From the doc linked previously... "A large reference or its enable parent that has the Initially Disabled flag can trigger the bugs." 

Also in the doc: "Alternatively, all large references of an affected cell could be moved underground out of view and new references replacing them could be added. These new reference wouldn't be large references. This is also a solution for large reference bugs triggered by the Initially Disabled flag in case it can not be removed."

I interpret this as somewhat contradictory in the doc (or at least very confusing) and also in some pf the back & forth posting on this topic. Please straighten me out. Here's the example 'fix' employed on BDA (let's just assume this is a LR with the bug verified; although, it was created specifically to prevent a DynDOLOD warning in the log and is likely not even an LR or LR with the bug):

How can "initially disabled" flag on REFR records be both the cause and a fix? Is it due to the plugin format (ESM/ESP/ESL)? I cannot find a clear explanation. Perhaps it's info learned since this fix was devised?

2. Said 'fix' exemplified in #1 seems to contain the data required for DynDOLOD to create it's own override, setting the dummy 'WorshipperActivator' for the -30000 buried object. Is this true? Specifically, setting "initially disabled" flag, z = -300000, and "PlayerRef" as enable parent? This info is alluded is in the posting chain I linked at top. Please debunk if I am misinterpreting.
3. How does the LR Fix MCM toggle fit into the timeline of our understanding and reaction to the LR bug problem? Is this a total workaround rather than using the example in #1? Should we even be doing what is shown in #1?
4. The LR workarounds doc isn't clear (to me) in how to employ, who should employ, or the exact result to expect from employing. Perhaps "just do it" applies and "you'll figure it out if you just do it"?

A good and fairly comprehensive explanation of plugin formats is posted on AFKMods, but it fails to address the context of why some plugins have changes --as in the LR mitigation in #1-- being made in the ESM rather than in a ESL/P. I have some educated guesses/assumptions, but some context here would be helpful, because it's not clearly explained anywhere that I have found (notwithstanding attempts to do so in various sources).

For the record, I want to add an explanation I got from @TechAngel85 on the "XESP - enable parent" dataType/column/descriptor/'chunk' with "PlayerRef" and flag "Set enable state to opposite of parent" ... clarity on such things is helpful:

"This permanently disables the reference because the parent is the player, and the player is always enabled. Since that flag is present, it'll never be enabled bc it only enables opposite of the reference (player)."

Done for now, thanks!

[sheson]

Some of the answers below to OP may be mute by now, so I keep them brief (and hope they match the more detailed discussions)

1. It is not a fix. It is a workaround to not have any large references with object LOD visible in the cell by moving them all underground. The bugs happen underground out of sight.
2. DynDOLOD does not set Initially Disabled or the opposite player enable parent on large references as those triggers the bugs. It removes them from UDRs if possible.
3. The setting fixes dynamic LOD for large references switching off when it needs to.
4. The first sentence explains what the workarounds do: counter texture flicker caused by the bugs. The page explains what needs to be done to enable the workarounds. Ask specific questions if anything is unclear. This is very WIP and things are not settled yet.

Non ESM plugins overwriting large reference records always trigger the bugs in that cell. Hence if the goal is to not trigger the bugs anything about it needs to be done with ESM plugins.

If you want to make a plugin that does not trigger large reference bugs follow these rules:

- Do not overwrite large references with a non ESM flagged plugin. (now dealt with by the large reference workarounds)
- Do not set Initially Disabled flag. (now dealt with by the large reference workarounds)
- Do not set an Enabled Parent that causes the reference to be initially disabled. (now dealt with by the large reference workarounds)
  (UDRs in ESM flagged plugins are fixed by DynDOLOD.esm by removing the flag and enable parent and setting a base record without a model)
- Do not move the reference out of its original cell. (DynDOLOD.esm moves it underground and adds a new reference in its new position)
- Large reference needs to use STAT and MSTT (with that unknown2 flag set) base records, do not change types of used base records of existing large references (will be dealt with in-game in the future)
- Do not change scale of the reference or the base records objects bounds to make it smaller than the generation threshold (dealt with by DynDOLOD.esm by setting threshold to 1 and fixing/reporting zero object bounds)

While the in-game large reference workarounds eliminate the texture flicker after the fact, it is best to make plugins do not trigger the bugs in the first place as best as possible.

[Mousetick]

20 minutes ago, z929669 said:

my main concern is if it could be creating a potential LR bug if it were in fact a LR. And what if it's not a LR?

Ok I think I'm understanding now. This particular reference is not a LR, so there's nothing to worry about.

If you want to find out if a reference is a LR or not, you can use these instructions.

Since it's not a LR, making it an UDR is equivalent to removing it in a clean, safe way. It could have been simply marked as Initially Disabled, but making it a full UDR ensures it can't be re-enabled, as long as it's not overwritten by another plugin.

27 minutes ago, z929669 said:

It seems to me that you are saying "right click on Skyrim.esm in the right-pane header of that record, and 'copy as override into' the LRFix.esm and change the z coord on that record in LRFix.esm". Thus, the record in LRFix.esm overrides the record in Skyrim.esm.

Yes, exactly.

28 minutes ago, z929669 said:

Then in #3, you are saying to recreate this record (with a different EditorID) using BadPlugin.esp as the source?

.... SO, we have two separate records now in LRFix.esm pointing to the same reference? (e.g., they will not align in xEdit right pane)

Not exactly, but your remark made me realize I went overboard with the manual fix, which is still valid for overridden LRs in general, but overkill for Initially Disabled LRs.

You don't need to duplicate the LR if it's simply Initially Disabled, burying the original LR underground is enough:

1. Create a master plugin that should not be overridden by anything else. Let's call it LRFix.esm.
2. Override the original LR (e.g. from Skyrim.esm) and move it underground in LRFix.esm. Practically speaking, make an ITM of the LR, using 'Copy as override into...' and change its Z coordinate to -30000. That is the only change made to the original LR. Don't set it Initially Disabled, don't set it Enable Parent Opposite Player.
3. If BadPlugin.esp is a patch plugin, remove the LR override from it, so that LRFix.esm can be the last override.

Before:
Skyrim.esm REFR ID1 places Big Rock at position X/Y/Z in Tamriel
BadPlugin.esp REFR ID1 places Big Rock at position X/Y/Z in Tamriel + Initially Disabled

After:
Skyrim.esm REFR ID1 places Big Rock at position X/Y/Z in Tamriel
LRFix.esm REFR ID1 places Big Rock at position X/Y/-30000 in Tamriel

Regarding the distinction between override/overwrite, this is subject to interpretation and context. I don't really make the distinction myself for plugins but your point is well taken. Hopefully the revised and shortened manual fix procedure above should be clearer.

xEdit uses 'Copy as override (with overwriting) into...' to overwrite an existing override  DynDOLOD's documentation and logs use the term 'overwritten' 

[Mousetick]

11 minutes ago, z929669 said:

So one would or would not create an entirely new record for this LR?

Yes one definitely ought to.

12 minutes ago, z929669 said:

I suspect not, and that one would simply recreate (copy with override into) the offending record in a new patch plugin (to accompany LRFix.esm) and modify with what I call "subrecord overrides" to effectively replace the offending subrecord values? (so basically a normal override-patching method to a record)?

No, doing so would perpetuate or re-create the overriden LR issue that we're supposed to fix

Example of a correct fix:

Before:
Skyrim.esm REFR ID1 places Big Rock scaled 100% at position X1/Y1/Z1 in Tamriel
BadPlugin.esp REFR ID1 places Big Rock scaled 150% at position X2/Y2/Z2 in Tamriel
^^^ This triggers an 'Overwritten large reference' warning from DynDOLOD

After:
Skyrim.esm REFR ID1 places Big Rock scaled 100% at position X1/Y1/Z1 in Tamriel
LRFix.esm REFR ID1 places Big Rock scaled 100% at position X1/Y1/-30000 in Tamriel
LRFix.esm REFR ID2 places Big Rock scaled 150% at position X2/Y2/Z2 in Tamriel

End Result in-game: 2 references ID1 & ID2
REFR ID1 places Big Rock scaled 100% at position X1/Y1/-30000 in Tamriel
REFR ID2 places Big Rock scaled 150% at position X2/Y2/Z2 in Tamriel

Note the 2 references are completely distinct of each other, they each have their own ID. One overrides the original LR to make it not visible, the other is a duplicate copy of the BadPlugin override, it's now separate from the original LR and doesn't bug it out. Reference ID2 is not an override of anything, it's an entirely new reference added to the world. There is now one more Big Rock object placed into the world than there was before.

Reworded complete procedure to remove ambiguities:

1. Create a master plugin that should not be overridden by anything else. Let's call it LRFix.esm.
2. Override the original LR (e.g. from Skyrim.esm) and move it underground in LRFix.esm. Practically speaking, make an ITM of the LR, using 'Copy as override into...' and change its Z coordinate to -30000. That is the only change made to the original LR. Don't set it Initially Disabled, don't set it Enable Parent Opposite Player.
3. Make a duplicate of the LR from BadPlugin.esp into LRFix.esm. Practically speaking, use 'Copy as new record into...' in xEdit, from BadPlugin.esp into LRFix.esm. This duplicate doesn't strictly need to be in LRFix.esm, it could be created into any other plugin, including BadPlugin.esp itself.
4. Remove the LR override from BadPlugin.esp, so that LRFix.esm can be the last override.

[Mousetick]

Finding Problematic Large References

Method 1

The easiest method by far is simply to run DynDOLOD in a pseudo "dry run" mode on the load order, let it do all the validation work, and consult its summary report at the end.

1. Disable DynDOLOD mod in MO2's left pane, if applicable.
2. Start DynDOLOD. Select all worldspaces.
3. Configure minimal and lowest settings required to run the validation and speed up LOD generation which will not be used. See screenshot below. Basically everything can be turned off, except Dynamic LOD.
4. Click OK. Ignore the 'Tree LOD generation not checked!' warning.
5. When DynDOLOD is finished, don't save anything, simply exit.
6. Review the Large References Bugs (if any) in the Summary Report automatically opened in web browser.
7. Re-enable DynDOLOD mod in MO2's left pane, if applicable.

At step #3 above, you may want to save the configuration into a preset for later use.

IMPORTANT: DynDOLOD does not report LR bugs that it can automatically fix in its summary report. If necessary, refer to DynDOLOD's normal and debug logs for further details, or use Method 2 below.

Method 2

Should not be necessary in most cases, except for checking LRs issues not detected or reported by DynDOLOD, or for determining if a reference is a LR for any reason.

Start xLODGen - Terrain LOD beta from your mod manager with these command-line options:

-SSE -edit

You'll then get the same UI as xEdit. Using xLODGen in edit mode for routine plugin editing is not recommended. Use only for LR checking purpose.

Determining whether a reference is a LR:

• The reference must be referenced by at least one other object: the 'Referenced by (n)' tab must be visible at the bottom of the right side.

• Click on the 'Referenced by (n)' tab to display the list of references.
• Check whether at least one reference has the WRLD signature:
  • YES: The reference is a LR.
  • NO: The reference is not a LR.

Following are a few examples and counter-examples of LRs to illustrate the above:

 

Edited December 15, 2022 by Mousetick
Updated with sheson's input

[Mousetick]

Warning: wall of text ahead.

1. It's not clear what you're showing in the xEdit screenshot, but if I'm understanding correctly, the BDA-DisableRefs.esm contains an hypothetical fix for a LR bug? The overwrite made by BDA-DisableRefs.esm doesn't fix anything, it would actually cause LR bugs if it overwrote a LR.

For sake of clarity, there are 2 types of plugin: master (as in ESM) or patch (as in ESP):

1. Master: plugin with .esm extension, or with .esl extension, or with .esp extension and ESM-flagged in the header.
2. Patch: plugin with .esp extension and not ESM-flagged in the header.

To avoid any ambiguity about ESP plugins, I'll only refer to master and patch plugins.

One cause of LR bugs is: a plugin, of any type (master or patch), overwrites a LR, or overwrites the Enable Parent of a LR, and sets it as Initially Disabled. Let's call it BadPlugin.esp.

The manual fix is:

1. Create a master plugin that should not be overwritten by anything else. Let's call it LRFix.esm.
2. Overwrite the original LR (e.g. from Skyrim.esm) and move it underground in LRFix.esm. Practically speaking, make an ITM of the LR and change its Z coordinate to -30000. That is the only change made to the original LR. Don't set it Initially Disabled, don't set it Enable Parent Opposite Player.
3. Make a duplicate of the LR from BadPlugin.esp into LRFix.esm. Practically speaking, use 'Copy as new record into...' in xEdit. This duplicate doesn't strictly need to be in LRFix.esm, it could be created into any other plugin. I prefer to keep it together into the same LRFix.esm plugin.
4. If BadPlugin.esp is a patch plugin, remove the LR overwrite from it, so that LRFix.esm can be the last overwrite.

That's it. What does this translate into for the game engine? You now have two references instead of one LR:

• The original LR, which is still enabled and in its original position/state, except it's moved vertically underground where it's not visible. It's still loaded/unloaded and processed normally by the LR system, without bugs (we hope).
• The duplicate, which is not a LR and therefore not processed by the LR system, that is initially disabled (or its Enable Parent is initially disabled).

The same manual solution works well, at least in my experience, to fix issues caused by a LR moved to another cell.

The manual solution works independently of DynDOLOD, whether it's used or not. But it's not foolproof: if another plugin overwrites the same LR and is loaded after LRFix.esm, it's back to square one - the new overwrite, depending on what changes it makes, could cause LR bugs again.

2. Setting "initially disabled" flag, z = -300000, and "PlayerRef" as enable parent is also known as UDR (UnDeleted Reference) for short. This is what xEdit does to references marked as Deleted in a plugin when autocleaning such plugin. This is not a fix for #1, per se. This is one set of conditions that DynDOLOD uses to determine if it can apply the fix automatically, the other conditions being last overwrite in load order, and overwrite made in a master plugin.

Making an UDR in a master plugin can be a solution to trigger DynDOLOD's autofix but it's dependent on DynDOLOD being used. On the other hand it can be more foolproof, as DynDOLOD.esm is always the last master in load order (doesn't change the situation).

 

---

About Enable Parent:

When making an UDR, xEdit adds an Enable Parent condition to the reference to prevent the initially disabled reference from being potentially re-enabled via a Papyrus script. Without it there would be a risk that a mod could re-enable the reference by script.

Making it opposite of Player guarantees that the Enable Parent condition is never satisfied.

[z929669]

1 hour ago, Mousetick said:

Warning: wall of text ahead.

1. It's not clear what you're showing in the xEdit screenshot, but if I'm understanding correctly, the BDA-DisableRefs.esm contains an hypothetical fix for a LR bug? The overwrite made by BDA-DisableRefs.esm doesn't fix anything, it would actually cause LR bugs if it overwrote a LR.

I'm glad taht my motive wasn't clear then, because you just confirmed my suspicion.

Yes, that was added as a placeholder of sorts to that ESM. It is a hypothetical fix for the LR bug but likely not a LR or a LR bug for that matter. Tech can explain more, but my main concern is if it could be creating a potential LR bug if it were in fact a LR. And what if it's not a LR? ... It sounds like the only purpose this particlular record could have in the BDA_DisableRefs plugin is to create a problem rather than solve one ... but IDK, honestly.

 

1 hour ago, Mousetick said:

The manual fix is:

1. Create a master plugin that should not be overwritten by anything else. Let's call it LRFix.esm.
2. Overwrite the original LR (e.g. from Skyrim.esm) and move it underground in LRFix.esm. Practically speaking, make an ITM of the LR and change its Z coordinate to -30000. That is the only change made to the original LR. Don't set it Initially Disabled, don't set it Enable Parent Opposite Player.
3. Make a duplicate of the LR from BadPlugin.esp into LRFix.esm. Practically speaking, use 'Copy as new record into...' in xEdit. This duplicate doesn't strictly need to be in LRFix.esm, it could be created into any other plugin. I prefer to keep it together into the same LRFix.esm plugin.

I'm probably misunderstanding, but ....

I think literally, so to me:

override = one has higher priority and 'wins' (non-destructive)

overwrite = one replaces and 'wins' (destructive)

Another potential point of misunderstanding (for me): A plugin record (xEdit left pane) has subrecords (or whatever the proper nomenclature; xEdit right pane). One must replicate the entire record into another plugin (or it must also exist in another plugin) for the subrecords to line up to make changes (I think of these as subrecord overrides inside a record override).

It seems to me that you are saying "right click on Skyrim.esm in the right-pane header of that record, and 'copy as override into' the LRFix.esm and change the z coord on that record in LRFix.esm". Thus, the record in LRFix.esm overrides the record in Skyrim.esm, and they are identical but for the subrecord override(s) in LRFix.esm.

Then in #3, you are saying to recreate this record (with a different EditorID) using BadPlugin.esp as the source?

.... SO, we have two separate records now in LRFix.esm pointing to the same reference? (e.g., they will not align in xEdit right pane)

(more on the rest later once I have made it over this cognitive hump )

1 hour ago, Mousetick said:

About Enable Parent:

When making an UDR, xEdit adds an Enable Parent condition to the reference to prevent the initially disabled reference from being potentially re-enabled via a Papyrus script. Without it there would be a risk that a mod could re-enable the reference by script.

Making it opposite of Player guarantees that the Enable Parent condition is never satisfied.

Good explanation. Thanks for taking the time

[z929669]

46 minutes ago, Mousetick said:

Ok I think I'm understanding now. This particular reference is not a LR, so there's nothing to worry about.

If you want to find out if a reference is a LR or not, you can use these instructions.

Since it's not a LR, making it an UDR is equivalent to removing it in a clean, safe way. It could have been simply marked as Initially Disabled, but making it a full UDR ensures it can't be re-enabled, as long as it's not overwritten by another plugin.

Yes, exactly.

Not exactly, but your remark made me realize I went overboard with the manual fix, which is still valid for overridden LRs in general, but overkill for Initially Disabled LRs.

You don't need to duplicate the LR if it's simply Initially Disabled, burying the original LR underground is enough:

1. Create a master plugin that should not be overridden by anything else. Let's call it LRFix.esm.
2. Override the original LR (e.g. from Skyrim.esm) and move it underground in LRFix.esm. Practically speaking, make an ITM of the LR, using 'Copy as override into...' and change its Z coordinate to -30000. That is the only change made to the original LR. Don't set it Initially Disabled, don't set it Enable Parent Opposite Player.
3. If BadPlugin.esp is a patch plugin, remove the LR override from it, so that LRFix.esm can be the last override.

Before:
Skyrim.esm REFR ID1 places Big Rock at position X/Y/Z in Tamriel
BadPlugin.esp REFR ID1 places Big Rock at position X/Y/Z in Tamriel + Initially Disabled

After:
Skyrim.esm REFR ID1 places Big Rock at position X/Y/Z in Tamriel
LRFix.esm REFR ID1 places Big Rock at position X/Y/-30000 in Tamriel

Regarding the distinction between override/overwrite, this is subject to interpretation and context. I don't really make the distinction myself for plugins but your point is well taken. Hopefully the revised and shortened manual fix procedure above should be clearer.

xEdit uses 'Copy as override (with overwriting) into...' to overwrite an existing override  DynDOLOD's documentation and logs use the term 'overwritten' 

The terminology clarification is appreciated. Nomenclature is a great point of misunderstanding in my experience, but most people tend to be fine with reading between the lines. I am not one of them. Sometimes I resolve simple misunderstandings related to this, and other times I'm the only person in the room with questions because of this

Thank you

The linked post instructions for determining LR is helpful as well.

---

Now... at the risk of wearing your patience further (in which case, please respond per your inclinations), I'd like to broach the more complicated "overridden LRs" that are not initially disabled. And per your first version instructions invoking my question:
 

Quote

 

Then in #3, you are saying to recreate this record (with a different EditorID) using BadPlugin.esp as the source?

.... SO, we have two separate records now in LRFix.esm pointing to the same reference? (e.g., they will not align in xEdit right pane)

 

So one would or would not create an entirely new record for this LR?

I suspect not, and that one would simply recreate (copy with override into) the offending record in a new patch plugin (to accompany LRFix.esm) and modify with what I call "subrecord overrides" to effectively replace the offending subrecord values? (so basically a normal override-patching method to a record)?

EDIT: on re-reading your original though, I guess I am thrown off by your idea of placing this into LRFix.esm (it must be a different record than the the original Skyrim.esm override, or they/it would be the same record in LRFix.esm). This makes me think that it is indeed a new record/EditorID pointing to the same reference, and I honestly don't have much personal XP do draw on here.

[Mousetick]

1 hour ago, z929669 said:

I guess I am thrown off by your idea of placing this into LRFix.esm

Perhaps. As mentioned in the instructions "This duplicate doesn't strictly need to be in LRFix.esm, it could be created into any other plugin, including BadPlugin.esp itself."

Creating new Temporary references in a master plugin is more efficient for the engine than adding them in a patch plugin. Temporary references added in a patch plugin are always loaded in memory whether they are used in active cell or not, like Persistent references. By contrast, Temporary references added in a master plugin are loaded in memory only on an as-needed basis, i.e. when they are used in an active cell, and unloaded otherwise.

That's why ESM-flagging entire patch plugins to become master plugins, in order to prevent LR bugs, can break mods due to bugs that were not noticeable before: Temporary references that used to be permanently loaded from the patch plugin, are now truly temporary.

[TechAngel85]

Just wanted to chime in to say that Mousetick is completely correct in how this is explained. It's exactly how I fix LRs since getting the info from Sheson. BDA was a bad example to use, initially, as those aren't really LRs, but Mousetick explained their purpose well.

[z929669]

Very good then. Thanks all for contributing your expertise on this subject.

We now have a topic to which we can point to for those wanting to better understand this issue and methods to fix it.

This will certainly help me to remember the relevant details as currently understood (I tend to forget things if I don't work with them routinely ... aging brain).

[sheson]

I added a couple more "rules" to the list.

[z929669]

19 hours ago, Mousetick said:

Ok I think I'm understanding now. This particular reference is not a LR, so there's nothing to worry about.

If you want to find out if a reference is a LR or not, you can use these instructions.

Since it's not a LR, making it an UDR is equivalent to removing it in a clean, safe way. It could have been simply marked as Initially Disabled, but making it a full UDR ensures it can't be re-enabled, as long as it's not overwritten by another plugin.

So employing your method to this BDA UDR record (Dragonborn is the master, and SMIM overrides the BDA UDR):

Is it a LR? (seems so, if I'm understanding that post correctly)

If so, could I "Copy as new record into ..." from SMIM into my patch per your more recent, elaborated-upon instruction?

EDIT: I am guessing I would also forward that record from BDA "as is" into my patch, right? That way, the UDR itself survives and the new reference essentially recreates what SMIM is doing?

[Mousetick]

7 hours ago, z929669 said:

Is it a LR? (seems so, if I'm understanding that post correctly)

This one is not a LR either. It's not that you're misunderstanding that post, rather that post was written quickly and is too vague/ambiguous, prone to confusion.

I'm going to redo the instructions for checking LRs and I'll post them here.

[sheson]

For the log messages run, only checking dynamic LOD should be enough, no need to check object LOD.

Note that the plan is to make the large reference workarounds the default.

xLODGen terrain LOD beta edit mode is the same thing as xEdit Extremely Experimental, just with the RNAM data not removed from the Tamriel worldspace record in Skyrim.esm and of course the changed LOD beta code.

[Mousetick]

7 hours ago, sheson said:

For the log messages run, only checking dynamic LOD should be enough, no need to check object LOD.

Thanks for the correction, I've amended the instructions accordingly.

7 hours ago, sheson said:

xLODGen terrain LOD beta edit mode is the same thing as xEdit Extremely Experimental, just with the RNAM data not removed from the Tamriel worldspace record in Skyrim.esm

Yes but to simplify I thought it better to advise always using xLODGen, since most of the Tamriel LRs are in Skyrim.esm.