DynDOLOD 3.00 Alpha 196

[Blackread]

1 hour ago, sheson said:

A question would be how the plugin was made ESM and how/if that affects navmeshes etc.

The mod RedBag's Rorikstead also contains texture(s) that have resolutions that are non power of 2, which I assume you must have fixed.

I used this script for the process: https://www.nexusmods.com/skyrimspecialedition/mods/40260 It applies the ESM flag and flags some NPCs as persistent. However, RedBag's Rorikstead does not add any new NPCs, so the script only applies the ESM flag.

I only have the lod texture fix from the mod page actually, I have not attempted to fix the other textures. Not really sure how I would go about doing so without messing the UV mapping.

[sheson]

1 hour ago, Blackread said:

I used this script for the process: https://www.nexusmods.com/skyrimspecialedition/mods/40260 It applies the ESM flag and flags some NPCs as persistent. However, RedBag's Rorikstead does not add any new NPCs, so the script only applies the ESM flag.

I only have the lod texture fix from the mod page actually, I have not attempted to fix the other textures. Not really sure how I would go about doing so without messing the UV mapping.

For plugins with navmeshes it might be necessary to involve CK but that is outside my expertise.

[Blackread]

1 hour ago, sheson said:

For plugins with navmeshes it might be necessary to involve CK but that is outside my expertise.

I don't think that's the whole truth here, as I have used the same script on many other plugins without issues, like all of the Great City/Town/Village mods by soldierofwar. But this falls very much outside my expertise too.

Two other mods that produced crashing when ESMified were Moon and Star and the 2020 Grand Paladin remake. Other users have reported similar issues with these mods too.

[Dez756]

5 hours ago, sheson said:

Do the same for any of the reported textures that are shown when the currently used NIF is highlighted in the asset browser.

Will do!  

Just to be sure, should I be running Dyndolod after making the foregoing changes before testing?  So far I haven't been.

[Dez756]

33 minutes ago, Dez756 said:

Do the same for any of the reported textures that are shown when the currently used NIF is highlighted in the asset browser.

Unlike the Meshes, I don't see a way to save the vanilla textures using the Asset Browser.  Am I missing something?

Edit:  Nevermind!  I think I figured it out.

Edited June 28, 2022 by Dez756

[Mousetick]

10 hours ago, Blackread said:

Anyway, I hope this information can help someone else experiencing the same crashes. If you have any questions I will try and answer them, assuming I get notifications for replies to this post.

No offense, but your analysis doesn't demonstrate or explain anything regarding the root cause of your crashes, and is not readily applicable to other users, because it's completely specific to your load order and it doesn't precisely identify the root cause.

All we learn from your analysis is that there is some interaction/conflict (maybe related to navmesh(es)) between some mods in your load order that causes a crash depending on their relative order.

ESMifying a mod changes its position in the plugin load order, and if that mod uses a BSA, it also changes the overwrite priority of its assets and resources in the load order. This is a critical detail that may be easy to overlook. If this mod conflicts with other mods in the load order, the conflicts are now resolved with a different priority with a different outcome. Unless manual patching is done in xEdit and/or manual extraction of BSAs is done to properly resolve those conflicts, the end results will vary in-game.

Another important detail that may easily be missed, and is a corollary to the previous, is that once the load order is modified, especially in the ESM range, the DynDOLOD output should be considered invalid and unsafe. It needs to be re-generated from scratch. This is crucial to ensure DynDOLOD.esm is properly synced up with the load order. When you ESMify a plugin without re-generating DynDOLOD, the previous DynDOLOD.esm doesn't account for the change. For example, it may overwrite references that should be marked persistent but are not because the outdated DynDOLOD.esm doesn't forward changes from the now ESMified plugin. Non-persistent references defined in an ESM are removed from memory when their parent cell is unloaded, causing unpredictable behavior or crashes when they're accessed.

You mention in your analysis that the crash stopped when you disabled a group of mods, including DynDOLOD Output. You didn't specify whether you had re-generated DynDOLOD after ESMifying and re-sorting your mods.

I've ESMified a number of mods (~20), using the script you referenced, including Moon and Star, without any issues. My current load order contains 668 plugins and I've yet to experience a game-breaking bug or crash that is not due to my own mistakes.

If I may offer some advice to help ensure a stable setup:

• Sort load order with LOOT. Use custom ordering rules only as a last resort.
• Systematically clean plugins with xEdit.
• Systematically check for plugins errors in xEdit.
• Systematically check for plugin conflicts and patch them in xEdit.
• Systematically check for conflicts among BSAs, and between BSAs and loose files. Enable 'Archives Parsing' experimental feature in MO2 if using MO2. Overwritten Papyrus scripts require particular attention.
• Re-generate DynDOLOD plugins whenever the load order changes, especially if it changes in the ESM range.
• Religiously pay attention to warnings in DynDOLOD's Summary Report.
• For extra care, review the warnings in DynDOLOD's log files. Some of them are not reported in the Summary Report but are still noteworthy and may require action.
• Fix as many found issues as possible depending on severity, by yourselves, or seek out solutions and patches elsewhere.
• Re-generate 'dynamic/automatic' patches, such as Synthesis or similar, whenever the load order changes.

Needless to say, this is not a simple matter of ESMifying plugin with an xEdit script. It's a lot of time-consuming work that requires some "expertise". So perhaps the best advice would be not to ESMify mods in the first place unless you understand the implications and are prepared to deal with them.

[Blackread]

2 hours ago, Mousetick said:

No offense, but your analysis doesn't demonstrate or explain anything regarding the root cause of your crashes, and is not readily applicable to other users, because it's completely specific to your load order and it doesn't precisely identify the root cause.

All we learn from your analysis is that there is some interaction/conflict (maybe related to navmesh(es)) between some mods in your load order that causes a crash depending on their relative order.

ESMifying a mod changes its position in the plugin load order, and if that mod uses a BSA, it also changes the overwrite priority of its assets and resources in the load order. This is a critical detail that may be easy to overlook. If this mod conflicts with other mods in the load order, the conflicts are now resolved with a different priority with a different outcome. Unless manual patching is done in xEdit and/or manual extraction of BSAs is done to properly resolve those conflicts, the end results will vary in-game.

Another important detail that may easily missed, and is a corollary of the previous, is that once the load order is modified, especially in the ESM range, the DynDOLOD output should be considered invalid and unsafe. It needs to be re-generated from scratch. This is crucial to ensure DynDOLOD.esm is properly synced up with the load order. When you ESMify a plugin without re-generating DynDOLOD, the previous DynDOLOD.esm doesn't account for the change. For example, it may overwrite references that should be marked persistent but are not because the outdated DynDOLOD.esm doesn't forward changes from the now ESMified plugin. Non-persistent references defined in an ESM are removed from memory when their parent cell is unloaded, causing unpredictable behavior or crashes when they're accessed.

You mention in your analysis that the crash stopped when you disabled a group of mods, including DynDOLOD Output. You didn't specify whether you had re-generated DynDOLOD after ESMifying and re-sorting your mods.

I've ESMified a number of mods (~20), using the script you referenced, including Moon and Star, without any issues. My current load order contains 668 plugins and I've yet to experience a game-breaking bug or crash that is not due to my own mistakes.

If I may offer some advice to help ensure a stable setup:

• Sort load order with LOOT. Use custom ordering rules only as a last resort.
• Systematically clean plugins with xEdit.
• Systematically check for plugins errors in xEdit.
• Systematically check for plugin conflicts and patch them in xEdit.
• Systematically check for conflicts among BSAs, and between BSAs and loose files. Enable 'Archives Parsing' experimental feature in MO2 if using MO2. Overwritten Papyrus scripts require particular attention.
• Re-generate DynDOLOD plugins whenever the load order changes, especially if it changes in the ESM range.
• Religiously pay attention to DynDOLOD's Summary Report and fix as many reported issues as possible.
• For extra care, review the warnings in DynDOLOD's log files. Some of them are not reported in the Summary Report but are still noteworthy and may require action.
• Fix as many found issues as possible depending on severity, by yourselves, or seek out solutions and patches elsewhere.

Needless to say, this is not a simple matter of ESMifying plugin with an xEdit script. It's a lot of time-consuming work that requires some "expertise". So perhaps the best advice would be not to ESMify mods in the first place unless you understand the implications and are prepared to deal with them.

In terms of the records edited by RedBag's Rorikstead, the relative order of the overwrites did not change between being an ESM or not. When ESM flagged the only plugins that were overwriting records from RedBag's Rorikstead were patches for RedBag's Rorikstead (non-ESM), that will overwrite the plugin regardless. In terms of assets, they are all packed in a bsa and all the overwrites are loose files, so no change there either.

The DynDOLOD Output I was testing this with was generated with the ESMified version of the plugin, so if anything, it was the non-ESM plugin that should have made it invalid and unsafe. Of course, removing the ESM flag did change some things in the overwrites, because the DynDOLOD.esm plugin overwrites disabled large references with its activator thingy, and these were reverted when the ESM flag was removed and the plugin moved above DynDOLOD.esm. But if this was the cause of the crash, just removing DynDOLOD output should have resolved it, but it didn't.

But prompted by your suspicion I decided to do some further testing. I placed the plugin as the last ESM plugin. This way, at least to my knowledge, the only thing that changes between adding and removing the ESM flag is the flag itself. With the flag the game crashed, without it didn't.

Next I took the original plugin from RedBag's Rorikstead, with the ITMs and deleted navmeshes intact, and added the ESM flag to that. This produced no crashing. Next I ran the plugin through the QuickAutoCleaner. This removed the ITMs, but left the deleted navmeshes. No crashing. Then I copied over the four edited navmesh records which I had separated into a new plugin here https://www.nexusmods.com/skyrimspecialedition/mods/69952, deleted the Navigation Mesh Info Map and regenerated it in CK. Now the game started to crash again.

Then I once again took the original plugin, cleaned ITMs, applied ESM flag, and this time used the separate plugin to overwrite the deleted navmeshes. These four navmesh records were edited in my LO by only Skyrim.esm, RedBag's Rorikstead.esp and my navmesh undeletion plugin. Now there were no crashes.

Putting all this together I would conclude that you were right in that it was a navmesh related issue. Not in terms of conflicts, but some undetermined latent issue with my edits to the deleted navmeshes that would only manifest if the plugin had ESM flag applied to it.

At this point I have pretty much tested all I can think of myself, and probably have reached the limits of my knowledge as well. I'll attach the broken plugin here in case you or someone else wishes to investigate this further. Thank you for your scepticism. It didn't make a big difference in the stability of my game as I had already resolved this issue in an alternative manner by separating the large ref edits into a plugin which I flagged as a master, but I did gain some more insight into the issue thanks to this more granular testing.

Thank you for the advice in your bullet points, but I am already doing all of those things regularly. But they without a doubt will help others reading this thread.

When it comes to Moon and Star though I can't say for sure what's going on with that one. My last experience with it was a couple years ago, around the time I published my ESMifier script. It very well may be that one was just a load order issue. I certainly wasn't as meticulous and thorough back then. Maybe I will revisit that subject some day again.

RedBag's Rorikstead.esp

Edited June 28, 2022 by Blackread

[sheson]

7 hours ago, Dez756 said:

Will do!  

Just to be sure, should I be running Dyndolod after making the foregoing changes before testing?  So far I haven't been.

No, for now just replace the files in the load order with the vanilla once which we assume should be fine.

[Mousetick]

2 hours ago, Blackread said:

Putting all this together I would conclude that you were right in that it was a navmesh related issue. Not in terms of conflicts, but some undetermined latent issue with my edits to the deleted navmeshes that would only manifest if the plugin had ESM flag applied to it.

Thanks for following up with all the details and clarifications. At this point it's pretty clear the issue has nothing to do with DynDOLOD and is specific to your navmesh patch so I'll keep my comments short:

• Editing navmeshes in CK is way beyond my knowledge so I can't help you there.
• As far as I understand, if the Navigation Mesh Info Map links to door/portal/passage references, they need to be persistent. Not being explicitly defined as persistent in the ESP may not be a problem, but could become one once the plugin is ESMified,  as they would now be truly temporary.

2 hours ago, Blackread said:

It didn't make a big difference in the stability of my game as I had already resolved this issue in an alternative manner by separating the large ref edits into a plugin which I flagged as a master

That's good advice and is actually much easier and safer than ESMifying whole plugins if the goal is to fix a few large references bugs rather than limiting the number of references.

[Blackread]

1 hour ago, Mousetick said:

Thanks for following up with all the details and clarifications. At this point it's pretty clear the issue has nothing to do with DynDOLOD and is specific to your navmesh patch so I'll keep my comments short:

• Editing navmeshes in CK is way beyond my knowledge so I can't help you there.
• As far as I understand, if the Navigation Mesh Info Map links to door/portal/passage references, they need to be persistent. Not being explicitly defined as persistent in the ESP may not be a problem, but could become one once the plugin is ESMified,  as they would now be truly temporary.

That's good advice and is actually much easier and safer than ESMifying whole plugins if the goal is to fix a few large references bugs rather than limiting the number of references.

Yes, I actually never meant to imply that this was a DynDOLOD issue. I mainly made a post here because there was a similar looking crash report that seemed to have been somewhat unresolved.

I can't take credit for the separate large ref plugin trick, as it's actually from the DynDOLOD Large Ref guide. But it is likely the least error prone solution in many cases. I often just ESMify the whole plugin out of laziness. xD

[Dez756]

3 hours ago, sheson said:

No, for now just replace the files in the load order with the vanilla once which we assume should be fine.

I replaced all five texture files shown for the icebergsmall02.nif in Asset Browser with vanilla textures, checking to make sure the path was correct in the MO2 Data tab.  Stil CTD'd at Riften Gate.  See crash log.

As a test, I again disabled the Dyndolod Output and traveled to the same spot.  No CTD.

Crash_2022_6_28_15-38-18.txt

[sheson]

46 minutes ago, Dez756 said:

I replaced all five texture files shown for the icebergsmall02.nif in Asset Browser with vanilla textures, checking to make sure the path was correct in the MO2 Data tab.  Stil CTD'd at Riften Gate.  See crash log.

As a test, I again disabled the Dyndolod Output and traveled to the same spot.  No CTD.

Crash_2022_6_28_15-38-18.txt 41.01 kB · 0 downloads

Seems we are not getting anywhere with this. Maybe time to do the steps explained under "Debug CTD caused by missing or invalid NIF" in the DynDOLOD-README.txt

First check/remove the object LOD mesh *.BTO file to rule them out.
If it still CTDs with all meshes removed, check if disabling DynDOLOD.esp and keeping DynDOLOD.esm makes a difference. In case it does, do the debug=true tests.

If it crashes with only DynDOLOD.esm, let me know for different suggestions.

[Dez756]

1 hour ago, sheson said:

If it still CTDs with all meshes removed, check if disabling DynDOLOD.esp and keeping DynDOLOD.esm makes a difference. In case it does, do the debug=true tests.

Still CTD with meshes removed.  No CTD with DynDOLOD.esp only disabled.

I'll try to get to the debug=true tests later.

[Dez756]

On 6/28/2022 at 5:17 PM, Dez756 said:

I'll try to get to the debug=true tests later.

Here's my Papyrus log with debug=true.  I don't see entries in the Papyrus log exactly like the example in ReadMe.  I tried to search for the Form ID of the last entry referencing DynDOLOD ([06/29/2022 - 03:54:57PM] [SHESON_DynDOLOD_LODObject < (BE028E70)>] [SHESON_DynDOLOD_Firstborn < (BE028E5A)>] Enable #2 [Form < (20004052)>] using dyndolod\lod\effects\fxfirewithembers01_dyndolod_lod.nif TRUE False TRUE) but SSEEdit doesn't seem to find anything (although I'm not sure I'm looking in the right spot - I've used SSEEdit for some things like cleaning, adding keywords or heel sounds but never searched for Form IDs).  See screen shot for the results of filter for BE028E70.  

I note that the entries in the log in the last second all seem to concern Creature Framework.  To test whether that was somehow involved, I removed CF and all associated mods and traveled (on a new game) to the same Riften Gate.  Still CTD'd.  Again, though this mod and its associates have been in my load order for literally years!

It's so weird that this seems to be the only place it CTDs as I had traveled to most of the other cities before I ran across this problem at Riften.

I'm sure this is getting frustrating for you!  Should I start (dread) removing mods and running everything (Xlodgen, TexGen, DynDOLOD) again? 

 

Papyrus.0.log

Edited June 29, 2022 by Dez756

[Dez756]

In case it helps, here's a screenshot right as the game CTD's.