DynDOLOD 3.00 Alpha 196

[sheson]

51 minutes ago, LordIceWolf said:

I found the source of the issue and just wanted to post it here in case someone else encounter the same issue. I had flagged the Whiterun Tree Plugin as esl and it appears to be the source of the problem. I reinstalled it as a regular esp and reran DynDOLOD and everything now works as it should.

And what would the name of the mod(s) and plugin(s) be?

[LordIceWolf]

1 hour ago, sheson said:

And what would the name of the mod(s) and plugin(s) be?

It's the optional file on this mod
https://www.nexusmods.com/skyrimspecialedition/mods/20396

The plugin is named Timber for Whiterun MystiriousDawn.esp

[Blackread]

I was wondering, since DynDOLOD already warns about overwritten large references and large references with the Initially Disabled flag, would it also be possible to add a warning about large references that have been moved outside their original cell? At least on my setup these kind of edits seem to fairly consistently trigger an issue where the data of the last plugin that places it in the origin cell is used instead of the override which moved it to another cell. This is especially true whenever the large ref system is enabled (uLargeRefLODGridSize > uGridsToLoad).

As an example, COTN Falkreath swaps the model of the Jarl's Longhouse and moves it up the hill (which is to say, into another exterior cell). If I have uLargeRefLODGridSize = uGridsToLoad it works fine, but if I enable large refs the longhouse ends up in its old position and using its old model. I can work around this with the disable & duplicate trick, but finding these issues would be a lot easier (and faster) if I had a list of all the records that need to be addressed, instead of having to manually check every overwritten large reference.

[Blackread]

Another question related to large references... I tried running the Skyrim SE - Generate Large References script on some plugins, and it sometimes results in non-existent form IDs ending up in the RNAM table. This seems to happen when there is an overwrite swapping the base object of a reference. The resulting form ID has the plugin index from the overwriting plugin, but the rest of the ID is from the plugin where the reference was introduced, resulting in a mismatched ID. Here's the log output when running the script on Water for ENB.esm and DLC2SolstheimWorld:

[00.00] Start: Applying script "Skyrim SE - Generate Large References"
Gathering large references added by Water for ENB.esm
Overwrite: [REFR:04021003] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDC6] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 15,16))
Overwrite: [REFR:04024528] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDAD] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 7,17))
Overwrite: [REFR:0402452C] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDCF] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 6,16))
Overwrite: [REFR:04024534] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of DLC2StoneWater02 [CELL:0400EDD0] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 5,16))
Overwrite: [REFR:04026BDC] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDA7] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 13,17))
Overwrite: [REFR:04026D14] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDAD] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 7,17))
Overwrite: [REFR:040274AE] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400ED85] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 14,18))
Overwrite: [REFR:0402868C] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of DLC2POIChest01 [CELL:0400ED48] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 9,20))
Overwrite: [REFR:04031957] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400ED83] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 16,18))
Overwrite: [REFR:04033EEF] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDA7] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 13,17))
Adding 10 large references to DLC2SolstheimWorld
[00.10] Done: Applying script "Skyrim SE - Generate Large References", Elapsed Time: 00.10

And here are some of the resulting entries in the RNAM table:

I was using the version of the script that comes bundled with xEdit 4.0.4

Edited July 11, 2022 by Blackread

[sheson]

5 hours ago, Blackread said:

I was wondering, since DynDOLOD already warns about overwritten large references and large references with the Initially Disabled flag, would it also be possible to add a warning about large references that have been moved outside their original cell? At least on my setup these kind of edits seem to fairly consistently trigger an issue where the data of the last plugin that places it in the origin cell is used instead of the override which moved it to another cell. This is especially true whenever the large ref system is enabled (uLargeRefLODGridSize > uGridsToLoad).

As an example, COTN Falkreath swaps the model of the Jarl's Longhouse and moves it up the hill (which is to say, into another exterior cell). If I have uLargeRefLODGridSize = uGridsToLoad it works fine, but if I enable large refs the longhouse ends up in its old position and using its old model. I can work around this with the disable & duplicate trick, but finding these issues would be a lot easier (and faster) if I had a list of all the records that need to be addressed, instead of having to manually check every overwritten large reference.

The reference is already reported as being overwritten, no?

What you describe is not really possible AFAIK I know. Check with More Informative Console which are the last plugins reported to change the reference / base record.

The engine enforces its own load order of plugins which can be different to plugins.txt.

1 hour ago, Blackread said:

Another question related to large references... I tried running the Skyrim SE - Generate Large References script on some plugins, and it sometimes results in non-existent form IDs ending up in the RNAM table. This seems to happen when there is an overwrite swapping the base object of a reference. The resulting form ID has the plugin index from the overwriting plugin, but the rest of the ID is from the plugin where the reference was introduced, resulting in a mismatched ID. Here's the log output when running the script on Water for ENB.esm and DLC2SolstheimWorld:

[00.00] Start: Applying script "Skyrim SE - Generate Large References"
Gathering large references added by Water for ENB.esm
Overwrite: [REFR:04021003] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDC6] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 15,16))
Overwrite: [REFR:04024528] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDAD] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 7,17))
Overwrite: [REFR:0402452C] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDCF] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 6,16))
Overwrite: [REFR:04024534] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of DLC2StoneWater02 [CELL:0400EDD0] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 5,16))
Overwrite: [REFR:04026BDC] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDA7] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 13,17))
Overwrite: [REFR:04026D14] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDAD] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 7,17))
Overwrite: [REFR:040274AE] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400ED85] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 14,18))
Overwrite: [REFR:0402868C] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of DLC2POIChest01 [CELL:0400ED48] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 9,20))
Overwrite: [REFR:04031957] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400ED83] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 16,18))
Overwrite: [REFR:04033EEF] (places MXCreekFlatLarge [STAT:FE005882] in GRUP Cell Temporary Children of [CELL:0400EDA7] (in DLC2SolstheimWorld "Solstheim" [WRLD:04000800] at 13,17))
Adding 10 large references to DLC2SolstheimWorld
[00.10] Done: Applying script "Skyrim SE - Generate Large References", Elapsed Time: 00.10

And here are some of the resulting entries in the RNAM table:

I was using the version of the script that comes bundled with xEdit 4.0.4

Test with latest version from https://github.com/TES5Edit/TES5Edit/blob/dev-4.1.5/Build/Edit Scripts/Skyrim SE - Generate Large References.pas

[Mousetick]

1 hour ago, sheson said:

The reference is already reported as being overwritten, no?

Not if the plugin(s) overwriting the reference are ESM or ESM-flagged ESP. In that case, DynDOLOD doesn't consider the overwrite buggy and doesn't warn about it.

An ESM moving a large reference to a different cell than where it is initially defined appears to trigger a different kind of bug than those described in https://dyndolod.info/Help/Large-References.

Consider for example:

• Skyrim.esm: adds REFR abc to Cell x1,y1 in Tamriel worldspace, includes it in Tamriel's RNAM table, so it's a large reference.
• Plugin.esm: overwrites REFR abc and moves it to Cell x2,y2.

From DynDOLOD's validation point of view, it's all good. What happens in-game however is something like this:

• Player approaches Cell x2,y2 by entering one of the near cells inside the uGrid encompassing Cell x2,y2: full model is loaded in Cell x2,y2, as expected.
• Player approaches Cell x2,y2 by entering one of the far cells inside the uLargeRefLODGridSize encompassing Cell x1,y1: full model is loaded in Cell x1,y1, which is the wrong one.

Basically the engine is given inconsistent information about the location of the large reference, the RNAM table being out of sync with the actual cell location of the REFR.

Hope my description is relatively clear and makes sense.

Thanks.

Addendum:

I should add that this 'RNAM out-of-sync with moved large reference issue' seems to arise from ESMifying ESP plugins, which don't have their own RNAM tables, or by creating an ESM plugin entirely in xEdit without manually generating the RNAM tables. If I'm not mistaken, such a situation would never normally arise in an ESM 'natively' constructed by CK, because CK is supposed to automatically generate the RNAM table? In that case, the CK-generated RNAM table for that ESM would reflect the move of the large reference to a different cell.

Edited July 11, 2022 by Mousetick
Added stuff I forgot

[sheson]

1 hour ago, Mousetick said:

Not if the plugin(s) overwriting the reference are ESM or ESM-flagged ESP. In that case, DynDOLOD doesn't consider the overwrite buggy and doesn't warn about it.

An ESM moving a large reference to a different cell than where it is initially defined appears to trigger a different kind of bug than those described in https://dyndolod.info/Help/Large-References.

Consider for example:

• Skyrim.esm: adds REFR abc to Cell x1,y1 in Tamriel worldspace, includes it in Tamriel's RNAM table, so it's a large reference.
• Plugin.esm: overwrites REFR abc and moves it to Cell x2,y2.

From DynDOLOD's validation point of view, it's all good. What happens in-game however is something like this:

• Player approaches Cell x2,y2 by entering one of the near cells inside the uGrid encompassing Cell x2,y2: full model is loaded in Cell x2,y2, as expected.
• Player approaches Cell x2,y2 by entering one of the far cells inside the uLargeRefLODGridSize encompassing Cell x1,y1: full model is loaded in Cell x1,y1, which is the wrong one.

Basically the engine is given inconsistent information about the location of the large reference, the RNAM table being out of sync with the actual cell location of the large REFR.

Hope my description is relatively clear and makes sense.

Thanks.

Addendum:

I should add that this 'RNAM out-of-sync with moved large reference issue' seems to arise from ESMifying ESP plugins, which don't have their own RNAM tables, or by creating an ESM plugin entirely in xEdit without manually generating the RNAM tables. If I'm not mistaken, such a situation would never normally arise in an ESM 'natively' constructed by CK, because CK is supposed to automatically generate the RNAM table? In that case, the CK-generated RNAM table for that ESM would reflect the move of the large reference to a different cell.

There is no mention in the post that the plugin has been ESM flagged, it seems to be an normal ESP based on the information provided.
Seems to me the plugin triggers lots of large reference bugs already.

The CK often adds all surrounding cells to the  RNAM table, not only the one the large reference is in. This is used for pre-loading references one cell ahead (Skyrim LE or if large references are off).

Be that as it may, the last plugin to overwrite anything is typically not ignored (unless the Ignore flag is set), so whatever base record  / model is used is typically defined by the last overwrite. 

[Mousetick]

29 minutes ago, sheson said:

There is no mention in the post that the plugin has been ESM flagged, it seems to be an normal ESP based on the information provided.

I'm pretty sure that's the case, based on my conversation with the OP elsewhere, and this information was omitted. The OP is attempting to fix the large reference bugs by ESMifying plugins. But of course it's not for me to say. I'll let the OP confirm whether they've ESM-flagged the plugin or not.

[sheson]

12 hours ago, LordIceWolf said:

It's the optional file on this mod
https://www.nexusmods.com/skyrimspecialedition/mods/20396

The plugin is named Timber for Whiterun MystiriousDawn.esp

I compacted the FormIDs and ESL flagged the plugin Timber for Whiterun MystiriousDawn.esp and removed "treereach, treepine, treeaspen," from the Ignore= line in DynDOLOD_SSE.ini and then generated LOD. The trees were copied from the child world to the parent word and LOD was generated for them as expected.

DynDOLOD_SSE_ChildworldCopies_Tamriel.txt contains the expected lines for the trees as well. For example:

[REFR:FE001D8F] (places TreePineForest02 [TREE:00018A02] in GRUP Cell Temporary Children of WhiterunPlainsDistrict04 [CELL:0001A276] (in WhiterunWorld "Whiterun" [WRLD:0001A26F] at 6,-2)) -> timberforwhiterunmystiriousdawnesp_000D8F_DynDOLOD_CHILD [REFR:084D18F5] ...

[sheson]

11 hours ago, Blackread said:

I was wondering, since DynDOLOD already warns about overwritten large references and large references with the Initially Disabled flag, would it also be possible to add a warning about large references that have been moved outside their original cell? At least on my setup these kind of edits seem to fairly consistently trigger an issue where the data of the last plugin that places it in the origin cell is used instead of the override which moved it to another cell. This is especially true whenever the large ref system is enabled (uLargeRefLODGridSize > uGridsToLoad).

As an example, COTN Falkreath swaps the model of the Jarl's Longhouse and moves it up the hill (which is to say, into another exterior cell). If I have uLargeRefLODGridSize = uGridsToLoad it works fine, but if I enable large refs the longhouse ends up in its old position and using its old model. I can work around this with the disable & duplicate trick, but finding these issues would be a lot easier (and faster) if I had a list of all the records that need to be addressed, instead of having to manually check every overwritten large reference.

With either 1.5.97 or 1.6.353, either with the plugin COTN - Falkreath.esp from https://www.nexusmods.com/skyrimspecialedition/mods/56731 as is or after ESM flagging it without generating RNAM data, then generating LOD, I always see the new model and also the large reference seems to behave normally, e. g. it loads past the uGridsToLoad and turns off beyond uLargeRefLODGridSize.

[sheson]

5 hours ago, Mousetick said:

• Player approaches Cell x2,y2 by entering one of the near cells inside the uGrid encompassing Cell x2,y2: full model is loaded in Cell x2,y2, as expected.
• Player approaches Cell x2,y2 by entering one of the far cells inside the uLargeRefLODGridSize encompassing Cell x1,y1: full model is loaded in Cell x1,y1, which is the wrong one.

Do you have an example plugin where the actual full model loads in a different position/cell other than the position/cell that the winning overwrite of the reference defines? With a new game of course.

[Mousetick]

1 hour ago, sheson said:

Do you have an example plugin where the actual full model loads in a different position/cell other than the position/cell that the winning overwrite of the reference defines?

I had one a while ago. But it was with a large reference to a PlaneMarker STAT, so "full model" doesn't really apply there. I believe it's the same or very similar problem though.

@Blackread describes a better example in details with a boat in Solitude in this topic.

[sheson]

1 hour ago, Mousetick said:

I had one a while ago. But it was with a large reference to a PlaneMarker STAT, so "full model" doesn't really apply there. I believe it's the same or very similar problem though.

@Blackread describes a better example in details with a boat in Solitude in this topic.

There has to be something else at work, too. Save games do save positions of things, so any tests need to be with new game.

I would remove everything else from the plugin and only keep involved records and only test with the minimum number of plugins. That should at least quickly narrow it down to what records/changes are required to have the problem and others are more likely to be able to replicate it.

[Blackread]

 

9 hours ago, sheson said:

Test with latest version from https://github.com/TES5Edit/TES5Edit/blob/dev-4.1.5/Build/Edit Scripts/Skyrim SE - Generate Large References.pas

I got the same result with the latest version. However, I noticed that if I loaded only the plugin (and its masters) I wanted to generate RNAM for, I got the right form IDs. Is this how the script is intended to be used? Although this only worked for plugins that only had a single master editing the same worldspace. If there were more the same thing happened.

3 hours ago, sheson said:

Do you have an example plugin where the actual full model loads in a different position/cell other than the position/cell that the winning overwrite of the reference defines? With a new game of course.

Sorry my initial post was a little light on details, I thought this was a known engine limitation. I'll give the full reproduction steps with COTN Falkreath here.

I did the same as you, took COTN Falkreath, flagged it as ESM without generating RNAM and generated LOD with DynDOLOD Alpha 95. Then this is how I can get the glitch to manifest:

1. From the main menu, coc whiterun.

2. After loading into the game I did the following console commands (to help with testing): tmm 1, tdetect, tgm.

3. Fast travel to Peak's Shade Tower, southeast from Falkreath.

4. Run to Falkreath. The route you take shouldn't matter, but I beelined over the cliffs.

5. Be greeted with this sight:

Spoiler

6. Run up the hill to inspect the place where the object should be, to find a gaping hole:

Spoiler

That's all of the steps. However, if you instead coc straight to falkreathexterior01 (be it from the main menu or after coc'ing to whiterun first), the object is in the right place.

Spoiler

On both occasions this is what the record looked like in xEdit:

Spoiler

On these tests I coc'd from the main menu, but the same bug occured on my full modlist when starting a new game via Alternate Perspective, so I don't think the manner of starting the game is relevant.

The LOD is generated and displayed correctly:

Spoiler

If you situate yourself a suitable distance south of falkreath you can get a situation where both the LOD and the full model are displayed:

Spoiler

CRC of the COTN - Falkreath.esp:

Spoiler

  - name: 'COTN - Falkreath.esp'
    clean:
      - crc: 0x80E3CC70
        util: 'SSEEdit v4.0.4'

When I took these screenshots I had uGridsToLoad = 5 and uLargeRefLODGridSize = 11. However, this time when I tested with uLargeRefLODGridSize = 5 the glitch still happened.

Active mods and plugins:

Spoiler

The game version was 1.5.97. Here are the modlist and plugins in a text file format, along with my ini files: https://loadorderlibrary.com/lists/falkreath-test

I also did a quick test with the original COTN Falkreath plugin without the ESM flag, and the glitch did not happen.

If there's anything I forgot let me know.

Edited July 11, 2022 by Blackread

[sheson]

2 hours ago, Blackread said:

 

I got the same result with the latest version. However, I noticed that if I loaded only the plugin (and its masters) I wanted to generate RNAM for, I got the right form IDs. Is this how the script is intended to be used? Although this only worked for plugins that only had a single master editing the same worldspace. If there were more the same thing happened.

Sorry my initial post was a little light on details, I thought this was a known engine limitation. I'll give the full reproduction steps with COTN Falkreath here.

I did the same as you, took COTN Falkreath, flagged it as ESM without generating RNAM and generated LOD with DynDOLOD Alpha 95. Then this is how I can get the glitch to manifest:

1. From the main menu, coc whiterun.

2. After loading into the game I did the following console commands (to help with testing): tmm 1, tdetect, tgm.

3. Fast travel to Peak's Shade Tower, southeast from Falkreath.

4. Run to Falkreath. The route you take shouldn't matter, but I beelined over the cliffs.

5. Be greeted with this sight:

  Reveal hidden contents

6. Run up the hill to inspect the place where the object should be, to find a gaping hole:

  Reveal hidden contents

That's all of the steps. However, if you instead coc straight to falkreathexterior01 (be it from the main menu or after coc'ing to whiterun first), the object is in the right place.

  Reveal hidden contents

On both occasions this is what the record looked like in xEdit:

  Reveal hidden contents

On these tests I coc'd from the main menu, but the same bug occured on my full modlist when starting a new game via Alternate Perspective, so I don't think the manner of starting the game is relevant.

The LOD is generated and displayed correctly:

  Reveal hidden contents

If you situate yourself a suitable distance south of falkreath you can get a situation where both the LOD and the full model are displayed:

  Reveal hidden contents

CRC of the COTN - Falkreath.esp:

  Reveal hidden contents

  - name: 'COTN - Falkreath.esp'
    clean:
      - crc: 0x80E3CC70
        util: 'SSEEdit v4.0.4'

When I took these screenshots I had uGridsToLoad = 5 and uLargeRefLODGridSize = 11. However, this time when I tested with uLargeRefLODGridSize = 5 the glitch still happened.

Active mods and plugins:

  Reveal hidden contents

The game version was 1.5.97. Here are the modlist and plugins in a text file format, along with my ini files: https://loadorderlibrary.com/lists/falkreath-test

I also did a quick test with the original COTN Falkreath plugin without the ESM flag, and the glitch did not happen.

If there's anything I forgot let me know.

Only load the plugin you want to update RNAMs for (xEdit should automatically load all required masters).
If it creates unresolved form IDs then, let me know a mod/plugin how to reproduce.

The problem with the wrong position/model does not happen after you generate RNAM for the plugin you flagged ESM?
E.g. the problem happens if a large reference is overwritten by an ESM without RNAM and the new cell / reference is not listed in any RNAM of any ESM plugin (OK that might be not trivial to test for your right now - I am just saying out loud the conditions)