DynDOLOD 3.00 Alpha 180

[sheson]

2 hours ago, Dez756 said:

Here are two crash logs from this morning.  Relevant objects look the same, probable call stack a bit different.

Crash_2022_6_27_6-8-54.txt 39.55 kB · 4 downloads Crash_2022_6_27_6-17-30.txt 40.88 kB · 3 downloads

OK, the crash logs seems to indicate a problem with a texture used by NIF that has a BStriShape called "IcebergSmall02:0"

Unfortunately the logs do not contain any hint about the involved *.DDS or *.NIF.

Typically the NIF filename will be IcebergSmall02.NIF (but it is totally possible for any random *.NIF to contain such a named BSTriShape as well).

Load all plugins in xEdit (so it loads all BSA as well).

Start the xEdit Asset Browser with CTRL+F3.

Enter IcebergSmall02.nif into the filter field at the top. For now we are only interested in exact matches. Check the listed Containers (BSA or Data) to the right. The first one listed "wins". Data means loose file. If the winning container is Data, use the MO2 right windows data tab to find t he loose file and its mod it comes form. If it is a BSA other than Skyrim - Meshes1.bsa the BSA filename should hint the mod it comes from.

Since the NIF seems to have a NiSourceTexture instead of a BSShaderTextureSet I do not expect it to be the vanilla NIF.

Click the result filename to highlight it. The status window at the bottom should list texture filenames under Assets used in ...
You can enter the texture filenames into the filter field again to find out if they come from the mod or another mod.

A quick troubleshooting test if we on the right path would be to temporarily disable any mod modifying that NIF so the one from the vanilla BSA is used instead to see if there is no more CTD.
Alternative you can right click and Save As the vanilla NIF from Skyrim - Meshes1.bsa and put it with the correct path (meshes\lansdcape\ice\IcebergSmall02.nif) into the MO2 overwrite folder, so it loads last and overwrites the version(s) included in any mods/BSA.

[Dez756]

8 hours ago, sheson said:

OK, the crash logs seems to indicate a problem with a texture used by NIF that has a BStriShape called "IcebergSmall02:0"

Unfortunately the logs do not contain any hint about the involved *.DDS or *.NIF.

Typically the NIF filename will be IcebergSmall02.NIF (but it is totally possible for any random *.NIF to contain such a named BSTriShape as well).

Load all plugins in xEdit (so it loads all BSA as well).

Start the xEdit Asset Browser with CTRL+F3.

Enter IcebergSmall02.nif into the filter field at the top. For now we are only interested in exact matches. Check the listed Containers (BSA or Data) to the right. The first one listed "wins". Data means loose file. If the winning container is Data, use the MO2 right windows data tab to find t he loose file and its mod it comes form. If it is a BSA other than Skyrim - Meshes1.bsa the BSA filename should hint the mod it comes from.

Since the NIF seems to have a NiSourceTexture instead of a BSShaderTextureSet I do not expect it to be the vanilla NIF.

Click the result filename to highlight it. The status window at the bottom should list texture filenames under Assets used in ...
You can enter the texture filenames into the filter field again to find out if they come from the mod or another mod.

A quick troubleshooting test if we on the right path would be to temporarily disable any mod modifying that NIF so the one from the vanilla BSA is used instead to see if there is no more CTD.
Alternative you can right click and Save As the vanilla NIF from Skyrim - Meshes1.bsa and put it with the correct path (meshes\lansdcape\ice\IcebergSmall02.nif) into the MO2 overwrite folder, so it loads last and overwrites the version(s) included in any mods/BSA.

Here're the results from xEdit Asset Browser - it seems to indicate that the mod with the problem nif is the USSEP?!?  I have the most recent 1.5.97 version if that.

One other thing - somehow right before I started on this modlist, the game auto updated to 1.6xxx.  I used this (Unofficial Skyrim Downloader at Skyrim Special Edition Nexus - Mods and Community (nexusmods.com)) to roll back to 1.5.97 and then started my current playthrough.  I'm wondering if there's a problem with that patch that's causing all these problems, which I didn't have before.  Any thoughts?

I so appreciate the time you're taking with me. I'm going to be away from my computer for a bit but I'll start on the troubleshooting you recommended when I get back in about an hour.

[sheson]

1 hour ago, Dez756 said:

Here're the results from xEdit Asset Browser - it seems to indicate that the mod with the problem nif is the USSEP?!?  I have the most recent 1.5.97 version if that.

One other thing - somehow right before I started on this modlist, the game auto updated to 1.6xxx.  I used this (Unofficial Skyrim Downloader at Skyrim Special Edition Nexus - Mods and Community (nexusmods.com)) to roll back to 1.5.97 and then started my current playthrough.  I'm wondering if there's a problem with that patch that's causing all these problems, which I didn't have before.  Any thoughts?

I so appreciate the time you're taking with me. I'm going to be away from my computer for a bit but I'll start on the troubleshooting you recommended when I get back in about an hour.

The first Container is Data, which means a loose file is winning. Use MO2 right window Data tab to drill down to the file and then check the Mod column to find out with mod contains the loose file.

[Dez756]

1 hour ago, sheson said:

The first Container is Data, which means a loose file is winning. Use MO2 right window Data tab to drill down to the file and then check the Mod column to find out with mod contains the loose file.

Sorry I misunderstood. 

The mesh is in Glorious Glaciers (Glorious Glaciers at Skyrim Special Edition Nexus - Mods and Community (nexusmods.com)).

I disabled that mod but CTD'd at the same place.  The log is attached.  

I saved the vanilla mesh from Skyrim - Meshes1.bsa and put it in and confirmed in the MO2 data tab that it was in the correct path in overwrite.  CTD in the same place. 

Second second log.

Crash_2022_6_27_19-20-36.txt Crash_2022_6_27_19-49-33.txt

[Blackread]

A couple days ago I was googling for information related to a crash I was experiencing, and I found a post made here earlier this month:

Having now solved this crash I thought I would make a reply here to provide some information that may help in the future, if someone else shows up with a similar crash.

To be exact, I was actually experiencing two different crashes. One with the same address as Joe9075 https://pastebin.com/QftfaANk and another at a different address https://pastebin.com/fFsvKSQs. However, these both reference pathing related things, so I suspect they are very much related. My crashes were predominantly happening upon fast traveling or shortly after arriving at my destination, but they could also happen when simply running around.

First I tried to solve this with the binary search method of disabling mods, and I did find a complete list of mods that, when enabled, caused the game to crash. They were as follows:
Praedy's Staves AIO
Salt and Wind - ApachiiSkyHairMale SE v1.2 Retexture
Prince and the Pauper
Simply Realistic Armor and Weapons
Grass Cache
DynDOLOD Output

If even one of these mods was enabled the game would crash, and with all of them disabled it was crash free. However, I found this very unbelievable, as I had used some of them for a long time, and how could a hair texture replacer cause a pathing related crash? So I tried another approach and started reverting my recent changes to my list. Quite soon I found out that the real cause of the crash was actually my collection of ESMified plugins, and eventually I narrowed it down to the plugin from RedBag's Rorikstead. I removed the ESM flag, and the crashing was gone.

As to why flagging this plugin as a master produced crashing, I have no idea. It seems that some plugins just don't like being ESM flagged, I have seen the same behavior in the past with a couple other mods too. Checking for errors and running the QuickAutoClean revealed nothing. Though the original plugin from RBR does have 31 ITMs and 4 deleted navmeshes, but I had already fixed these for my modified version.
  - name: 'RedBag''s Rorikstead.esp'
    dirty:
      - <<: *reqManualFix
        crc: 0x471AAB46
        util: '[SSEEdit v4.0.4](https://www.nexusmods.com/skyrimspecialedition/mods/164)'
        itm: 31
        nav: 4
    dirty:
      - <<: *reqManualFix
        crc: 0xF229C160
        util: '[SSEEdit v4.0.4](https://www.nexusmods.com/skyrimspecialedition/mods/164)'
        nav: 4

Anyway, I hope this information can help someone else experiencing the same crashes. If you have any questions I will try and answer them, assuming I get notifications for replies to this post.

Edited June 28, 2022 by Blackread

[sheson]

5 hours ago, Dez756 said:

Sorry I misunderstood. 

The mesh is in Glorious Glaciers (Glorious Glaciers at Skyrim Special Edition Nexus - Mods and Community (nexusmods.com)).

I disabled that mod but CTD'd at the same place.  The log is attached.  

I saved the vanilla mesh from Skyrim - Meshes1.bsa and put it in and confirmed in the MO2 data tab that it was in the correct path in overwrite.  CTD in the same place. 

Second second log.

Crash_2022_6_27_19-20-36.txt 48.88 kB · 1 download Crash_2022_6_27_19-49-33.txt 47.67 kB · 1 download

Do the same for any of the reported textures that are shown when the currently used NIF is highlighted in the asset browser.

[sheson]

1 hour ago, Blackread said:

A couple days ago I was googling for information related to a crash I was experiencing, and I found a post made here earlier this month:

Having now solved this crash I thought I would make a reply here to provide some information that may help in the future, if someone else shows up with a similar crash.

To be exact, I was actually experiencing two different crashes. One with the same address as Joe9075 https://pastebin.com/QftfaANk and another at a different address https://pastebin.com/fFsvKSQs. However, these both reference pathing related things, so I suspect they are very much related. My crashes were predominantly happening upon fast traveling or shortly after arriving at my destination, but they could also happen when simply running around.

First I tried to solve this with the binary search method of disabling mods, and I did find a complete list of mods that, when enabled, caused the game to crash. They were as follows:
Praedy's Staves AIO
Salt and Wind - ApachiiSkyHairMale SE v1.2 Retexture
Prince and the Pauper
Simply Realistic Armor and Weapons
Grass Cache
DynDOLOD Output

If even one of these mods was enabled the game would crash, and with all of them disabled it was crash free. However, I found this very unbelievable, as I had used some of them for a long time, and how could a hair texture replacer cause a pathing related crash? So I tried another approach and started reverting my recent changes to my list. Quite soon I found out that the real cause of the crash was actually my collection of ESMified plugins, and eventually I narrowed it down to the plugin from RedBag's Rorikstead. I removed the ESM flag, and the crashing was gone.

As to why flagging this plugin as a master produced crashing, I have no idea. It seems that some plugins just don't like being ESM flagged, I have seen the same behavior in the past with a couple other mods too. Checking for errors and running the QuickAutoClean revealed nothing. Though the original plugin from RBR does have 31 ITMs and 4 deleted navmeshes, but I had already fixed these for my modified version.
  - name: 'RedBag''s Rorikstead.esp'
    dirty:
      - <<: *reqManualFix
        crc: 0x471AAB46
        util: '[SSEEdit v4.0.4](https://www.nexusmods.com/skyrimspecialedition/mods/164)'
        itm: 31
        nav: 4
    dirty:
      - <<: *reqManualFix
        crc: 0xF229C160
        util: '[SSEEdit v4.0.4](https://www.nexusmods.com/skyrimspecialedition/mods/164)'
        nav: 4

Anyway, I hope this information can help someone else experiencing the same crashes. If you have any questions I will try and answer them, assuming I get notifications for replies to this post.

A question would be how the plugin was made ESM and how/if that affects navmeshes etc.

The mod RedBag's Rorikstead also contains texture(s) that have resolutions that are non power of 2, which I assume you must have fixed.

[Blackread]

1 hour ago, sheson said:

A question would be how the plugin was made ESM and how/if that affects navmeshes etc.

The mod RedBag's Rorikstead also contains texture(s) that have resolutions that are non power of 2, which I assume you must have fixed.

I used this script for the process: https://www.nexusmods.com/skyrimspecialedition/mods/40260 It applies the ESM flag and flags some NPCs as persistent. However, RedBag's Rorikstead does not add any new NPCs, so the script only applies the ESM flag.

I only have the lod texture fix from the mod page actually, I have not attempted to fix the other textures. Not really sure how I would go about doing so without messing the UV mapping.

[sheson]

1 hour ago, Blackread said:

I used this script for the process: https://www.nexusmods.com/skyrimspecialedition/mods/40260 It applies the ESM flag and flags some NPCs as persistent. However, RedBag's Rorikstead does not add any new NPCs, so the script only applies the ESM flag.

I only have the lod texture fix from the mod page actually, I have not attempted to fix the other textures. Not really sure how I would go about doing so without messing the UV mapping.

For plugins with navmeshes it might be necessary to involve CK but that is outside my expertise.

[Blackread]

1 hour ago, sheson said:

For plugins with navmeshes it might be necessary to involve CK but that is outside my expertise.

I don't think that's the whole truth here, as I have used the same script on many other plugins without issues, like all of the Great City/Town/Village mods by soldierofwar. But this falls very much outside my expertise too.

Two other mods that produced crashing when ESMified were Moon and Star and the 2020 Grand Paladin remake. Other users have reported similar issues with these mods too.

[Dez756]

5 hours ago, sheson said:

Do the same for any of the reported textures that are shown when the currently used NIF is highlighted in the asset browser.

Will do!  

Just to be sure, should I be running Dyndolod after making the foregoing changes before testing?  So far I haven't been.

[Dez756]

33 minutes ago, Dez756 said:

Do the same for any of the reported textures that are shown when the currently used NIF is highlighted in the asset browser.

Unlike the Meshes, I don't see a way to save the vanilla textures using the Asset Browser.  Am I missing something?

Edit:  Nevermind!  I think I figured it out.

Edited June 28, 2022 by Dez756

[Mousetick]

10 hours ago, Blackread said:

Anyway, I hope this information can help someone else experiencing the same crashes. If you have any questions I will try and answer them, assuming I get notifications for replies to this post.

No offense, but your analysis doesn't demonstrate or explain anything regarding the root cause of your crashes, and is not readily applicable to other users, because it's completely specific to your load order and it doesn't precisely identify the root cause.

All we learn from your analysis is that there is some interaction/conflict (maybe related to navmesh(es)) between some mods in your load order that causes a crash depending on their relative order.

ESMifying a mod changes its position in the plugin load order, and if that mod uses a BSA, it also changes the overwrite priority of its assets and resources in the load order. This is a critical detail that may be easy to overlook. If this mod conflicts with other mods in the load order, the conflicts are now resolved with a different priority with a different outcome. Unless manual patching is done in xEdit and/or manual extraction of BSAs is done to properly resolve those conflicts, the end results will vary in-game.

Another important detail that may easily be missed, and is a corollary to the previous, is that once the load order is modified, especially in the ESM range, the DynDOLOD output should be considered invalid and unsafe. It needs to be re-generated from scratch. This is crucial to ensure DynDOLOD.esm is properly synced up with the load order. When you ESMify a plugin without re-generating DynDOLOD, the previous DynDOLOD.esm doesn't account for the change. For example, it may overwrite references that should be marked persistent but are not because the outdated DynDOLOD.esm doesn't forward changes from the now ESMified plugin. Non-persistent references defined in an ESM are removed from memory when their parent cell is unloaded, causing unpredictable behavior or crashes when they're accessed.

You mention in your analysis that the crash stopped when you disabled a group of mods, including DynDOLOD Output. You didn't specify whether you had re-generated DynDOLOD after ESMifying and re-sorting your mods.

I've ESMified a number of mods (~20), using the script you referenced, including Moon and Star, without any issues. My current load order contains 668 plugins and I've yet to experience a game-breaking bug or crash that is not due to my own mistakes.

If I may offer some advice to help ensure a stable setup:

• Sort load order with LOOT. Use custom ordering rules only as a last resort.
• Systematically clean plugins with xEdit.
• Systematically check for plugins errors in xEdit.
• Systematically check for plugin conflicts and patch them in xEdit.
• Systematically check for conflicts among BSAs, and between BSAs and loose files. Enable 'Archives Parsing' experimental feature in MO2 if using MO2. Overwritten Papyrus scripts require particular attention.
• Re-generate DynDOLOD plugins whenever the load order changes, especially if it changes in the ESM range.
• Religiously pay attention to warnings in DynDOLOD's Summary Report.
• For extra care, review the warnings in DynDOLOD's log files. Some of them are not reported in the Summary Report but are still noteworthy and may require action.
• Fix as many found issues as possible depending on severity, by yourselves, or seek out solutions and patches elsewhere.
• Re-generate 'dynamic/automatic' patches, such as Synthesis or similar, whenever the load order changes.

Needless to say, this is not a simple matter of ESMifying plugin with an xEdit script. It's a lot of time-consuming work that requires some "expertise". So perhaps the best advice would be not to ESMify mods in the first place unless you understand the implications and are prepared to deal with them.

[Blackread]

2 hours ago, Mousetick said:

No offense, but your analysis doesn't demonstrate or explain anything regarding the root cause of your crashes, and is not readily applicable to other users, because it's completely specific to your load order and it doesn't precisely identify the root cause.

All we learn from your analysis is that there is some interaction/conflict (maybe related to navmesh(es)) between some mods in your load order that causes a crash depending on their relative order.

ESMifying a mod changes its position in the plugin load order, and if that mod uses a BSA, it also changes the overwrite priority of its assets and resources in the load order. This is a critical detail that may be easy to overlook. If this mod conflicts with other mods in the load order, the conflicts are now resolved with a different priority with a different outcome. Unless manual patching is done in xEdit and/or manual extraction of BSAs is done to properly resolve those conflicts, the end results will vary in-game.

Another important detail that may easily missed, and is a corollary of the previous, is that once the load order is modified, especially in the ESM range, the DynDOLOD output should be considered invalid and unsafe. It needs to be re-generated from scratch. This is crucial to ensure DynDOLOD.esm is properly synced up with the load order. When you ESMify a plugin without re-generating DynDOLOD, the previous DynDOLOD.esm doesn't account for the change. For example, it may overwrite references that should be marked persistent but are not because the outdated DynDOLOD.esm doesn't forward changes from the now ESMified plugin. Non-persistent references defined in an ESM are removed from memory when their parent cell is unloaded, causing unpredictable behavior or crashes when they're accessed.

You mention in your analysis that the crash stopped when you disabled a group of mods, including DynDOLOD Output. You didn't specify whether you had re-generated DynDOLOD after ESMifying and re-sorting your mods.

I've ESMified a number of mods (~20), using the script you referenced, including Moon and Star, without any issues. My current load order contains 668 plugins and I've yet to experience a game-breaking bug or crash that is not due to my own mistakes.

If I may offer some advice to help ensure a stable setup:

• Sort load order with LOOT. Use custom ordering rules only as a last resort.
• Systematically clean plugins with xEdit.
• Systematically check for plugins errors in xEdit.
• Systematically check for plugin conflicts and patch them in xEdit.
• Systematically check for conflicts among BSAs, and between BSAs and loose files. Enable 'Archives Parsing' experimental feature in MO2 if using MO2. Overwritten Papyrus scripts require particular attention.
• Re-generate DynDOLOD plugins whenever the load order changes, especially if it changes in the ESM range.
• Religiously pay attention to DynDOLOD's Summary Report and fix as many reported issues as possible.
• For extra care, review the warnings in DynDOLOD's log files. Some of them are not reported in the Summary Report but are still noteworthy and may require action.
• Fix as many found issues as possible depending on severity, by yourselves, or seek out solutions and patches elsewhere.

Needless to say, this is not a simple matter of ESMifying plugin with an xEdit script. It's a lot of time-consuming work that requires some "expertise". So perhaps the best advice would be not to ESMify mods in the first place unless you understand the implications and are prepared to deal with them.

In terms of the records edited by RedBag's Rorikstead, the relative order of the overwrites did not change between being an ESM or not. When ESM flagged the only plugins that were overwriting records from RedBag's Rorikstead were patches for RedBag's Rorikstead (non-ESM), that will overwrite the plugin regardless. In terms of assets, they are all packed in a bsa and all the overwrites are loose files, so no change there either.

The DynDOLOD Output I was testing this with was generated with the ESMified version of the plugin, so if anything, it was the non-ESM plugin that should have made it invalid and unsafe. Of course, removing the ESM flag did change some things in the overwrites, because the DynDOLOD.esm plugin overwrites disabled large references with its activator thingy, and these were reverted when the ESM flag was removed and the plugin moved above DynDOLOD.esm. But if this was the cause of the crash, just removing DynDOLOD output should have resolved it, but it didn't.

But prompted by your suspicion I decided to do some further testing. I placed the plugin as the last ESM plugin. This way, at least to my knowledge, the only thing that changes between adding and removing the ESM flag is the flag itself. With the flag the game crashed, without it didn't.

Next I took the original plugin from RedBag's Rorikstead, with the ITMs and deleted navmeshes intact, and added the ESM flag to that. This produced no crashing. Next I ran the plugin through the QuickAutoCleaner. This removed the ITMs, but left the deleted navmeshes. No crashing. Then I copied over the four edited navmesh records which I had separated into a new plugin here https://www.nexusmods.com/skyrimspecialedition/mods/69952, deleted the Navigation Mesh Info Map and regenerated it in CK. Now the game started to crash again.

Then I once again took the original plugin, cleaned ITMs, applied ESM flag, and this time used the separate plugin to overwrite the deleted navmeshes. These four navmesh records were edited in my LO by only Skyrim.esm, RedBag's Rorikstead.esp and my navmesh undeletion plugin. Now there were no crashes.

Putting all this together I would conclude that you were right in that it was a navmesh related issue. Not in terms of conflicts, but some undetermined latent issue with my edits to the deleted navmeshes that would only manifest if the plugin had ESM flag applied to it.

At this point I have pretty much tested all I can think of myself, and probably have reached the limits of my knowledge as well. I'll attach the broken plugin here in case you or someone else wishes to investigate this further. Thank you for your scepticism. It didn't make a big difference in the stability of my game as I had already resolved this issue in an alternative manner by separating the large ref edits into a plugin which I flagged as a master, but I did gain some more insight into the issue thanks to this more granular testing.

Thank you for the advice in your bullet points, but I am already doing all of those things regularly. But they without a doubt will help others reading this thread.

When it comes to Moon and Star though I can't say for sure what's going on with that one. My last experience with it was a couple years ago, around the time I published my ESMifier script. It very well may be that one was just a load order issue. I certainly wasn't as meticulous and thorough back then. Maybe I will revisit that subject some day again.

RedBag's Rorikstead.esp

Edited June 28, 2022 by Blackread

[sheson]

7 hours ago, Dez756 said:

Will do!  

Just to be sure, should I be running Dyndolod after making the foregoing changes before testing?  So far I haven't been.

No, for now just replace the files in the load order with the vanilla once which we assume should be fine.