Performance issue maybe related to LOD

[w740su]

I recently redone my landscapes(changed some textures, switched from Majestic Mountains to Majestic Mountains Complex Material, and created my own grass patch), and after generating all the lods(grass cache, then xlodgen, then texture gen, and finally dyndolod), and I occasionally will have huge framerate drops(50+ to 10+) when looking at the mountains. This frame drop only happens when switching cells, but I cannot find out how to reproduce it. When the frame rate drops, the GPU usage will hit 100% with only about half of its power draw(RivaTuner's monitoring), in both RivaTuner and Task Manager the VRAM is not full, everything is on an NVME SSD and the drawcall isn't particularly high. And if I made a save immidiately and reboot the game, the framerate will be normal. The reason I think this could be related to the LOD is that I found I could reduce the frequency of this kind of frame drop by lowering the LOD settings in BethINI. I need some help about what I should check to find out what is causing this performance issue.

Thanks in advance!

[sheson]

53 minutes ago, w740su said:

I recently redone my landscapes(changed some textures, switched from Majestic Mountains to Majestic Mountains Complex Material, and created my own grass patch), and after generating all the lods(grass cache, then xlodgen, then texture gen, and finally dyndolod), and I occasionally will have huge framerate drops(50+ to 10+) when looking at the mountains. This frame drop only happens when switching cells, but I cannot find out how to reproduce it. When the frame rate drops, the GPU usage will hit 100% with only about half of its power draw(RivaTuner's monitoring), in both RivaTuner and Task Manager the VRAM is not full, everything is on an NVME SSD and the drawcall isn't particularly high. And if I made a save immidiately and reboot the game, the framerate will be normal. The reason I think this could be related to the LOD is that I found I could reduce the frequency of this kind of frame drop by lowering the LOD settings in BethINI. I need some help about what I should check to find out what is causing this performance issue.

Thanks in advance!

Read https://dyndolod.info/Official-DynDOLOD-Support-Forum#Post-Logs which DynDOLOD log and debug log to upload when making posts.

Read https://dyndolod.info/Official-DynDOLOD-Support-Forum#Rudimentary-Troubleshooting to narrow down the problem.

Read https://dyndolod.info/FAQ#Performance

Read https://dyndolod.info/FAQ#Long-running-time-or-output-several-GB-in-file-size

Performance issue when switching cells typically have to do with loading full models/textures for the cells.

[w740su]

I have done some testing in the past days. First the problem doesn't seem to happen without DynDOLOD's output. Without the DynDOLOD output the GPU power draw is constantly at over 300W but when the frame drop happens it only draws less than 180W. Also, the frame drop only happens after I fast travel for like 5-6 times consecutively, and only when I am looking at the mountains/trees(even when there is a tiny part of mountain/tree on screen), frame rate drops to around 10 fps. If it is caused by loading full models/textures, then I think it is most likely caused by Nature of The Wild Lands - 3D LODs. There was a weird warning I ignored last time I generated LOD: a small tree was so small that got ignored by TexGen but then DynDOLOD was reporting not finding its textures. 

I decided to regenerate LODs to get some newer logs, but this time I had a range check error when DynDOLOD is calculating occlusion for DLC2SolstheimWorld. Here are the logs: https://ufile.io/dqefrg5r . Some of the logs are from my last generation(March 7th), and in between I only downloaded/updated some dialogue mods and animation mods.

[sheson]

3 hours ago, w740su said:

I have done some testing in the past days. First the problem doesn't seem to happen without DynDOLOD's output. Without the DynDOLOD output the GPU power draw is constantly at over 300W but when the frame drop happens it only draws less than 180W. Also, the frame drop only happens after I fast travel for like 5-6 times consecutively, and only when I am looking at the mountains/trees(even when there is a tiny part of mountain/tree on screen), frame rate drops to around 10 fps. If it is caused by loading full models/textures, then I think it is most likely caused by Nature of The Wild Lands - 3D LODs. There was a weird warning I ignored last time I generated LOD: a small tree was so small that got ignored by TexGen but then DynDOLOD was reporting not finding its textures. 

I decided to regenerate LODs to get some newer logs, but this time I had a range check error when DynDOLOD is calculating occlusion for DLC2SolstheimWorld. Here are the logs: https://ufile.io/dqefrg5r . Some of the logs are from my last generation(March 7th), and in between I only downloaded/updated some dialogue mods and animation mods.

Test if DynDOLOD 3 Alpha-168 fixes the range check error.

The mod Nature of The Wild Lands - 3D LODs does not contain any full models and no textures.

Determine if disabling terrain + object LOD with tll as explained in https://dyndolod.info/Official-DynDOLOD-Support-Forum#Rudimentary-Troubleshooting makes a difference. If that is the case, then:

Some of 3D LOD models from Nature of The Wild Lands - 3D LODs are huge and reported by the log as explained by https://dyndolod.info/FAQ#Long-running-time-or-output-several-GB-in-file-size. There also seem 3D tree LOD models for NOTW with less file size in Nature of the Wild Lands - forest and trees improvement mod.
Since I never tested any of these mods myself I can not say anything about their quality myself. In any case, I suggest to test the performance impact of the 3D tree LOD by generating with standard tree LOD once for comparison.

Since it seems you are also generating grass LOD, be vary about the full grass/cache density, too. I would also suggest test performance without grass LOD.

[w740su]

Thanks for the reply. Sorry I forgot to check the DynDOLOD version and my DynDOLOD version was still v167. I am using v169 now and there's no range check error.

Following the Rudimentary Troubleshooting page, using tll will only delay the performance issue from ~5 to ~15 fast travels. I also tried generating a DynDOLOD without having the terrain LOD in the mod list and it also only delay the issue to ~15 fast travels. And If I load the save without anything in the DynDOLOD output folder game will CTD after ~15 fast travels.

The good news is I finally find something that is more closely related to the frame drops. In one testing run, I found some rocks from Majestic Mountains Complex Material not loading properly. Here are some screenshots. Sorry for using phone cameras for the screenshots. During the testing, whenever I have this rock(rockpilel01.nif, BaseID 000226BA, RefID 000A523C) in sight, the GPU would stop working at full power(screenshot1), and after I disabled it, I gained some FPS back(screenshot2). However, I am not able to replicate this scene. After some minutes disabling and enabling it back and forth, I switched to the MO2 window to check which mod the mesh is from and then alt-tab-ed back, everything then seemed to be loading properly and GPU drew over 300W(screenshot3). One notable thing about the rock(rockpilel01.nif) is that it also has a lot of warnings of textures not matching in the DynDOLOD summary although I checked all the textures are from either Majestic Mountains Complex Material or Majestic Landscapes.

DynDOLOD Summary.zip

[sheson]

3 hours ago, w740su said:

Thanks for the reply. Sorry I forgot to check the DynDOLOD version and my DynDOLOD version was still v167. I am using v169 now and there's no range check error.

Following the Rudimentary Troubleshooting page, using tll will only delay the performance issue from ~5 to ~15 fast travels. I also tried generating a DynDOLOD without having the terrain LOD in the mod list and it also only delay the issue to ~15 fast travels. And If I load the save without anything in the DynDOLOD output folder game will CTD after ~15 fast travels.

The good news is I finally find something that is more closely related to the frame drops. In one testing run, I found some rocks from Majestic Mountains Complex Material not loading properly. Here are some screenshots. Sorry for using phone cameras for the screenshots. During the testing, whenever I have this rock(rockpilel01.nif, BaseID 000226BA, RefID 000A523C) in sight, the GPU would stop working at full power(screenshot1), and after I disabled it, I gained some FPS back(screenshot2). However, I am not able to replicate this scene. After some minutes disabling and enabling it back and forth, I switched to the MO2 window to check which mod the mesh is from and then alt-tab-ed back, everything then seemed to be loading properly and GPU drew over 300W(screenshot3). One notable thing about the rock(rockpilel01.nif) is that it also has a lot of warnings of textures not matching in the DynDOLOD summary although I checked all the textures are from either Majestic Mountains Complex Material or Majestic Landscapes.

DynDOLOD Summary.zip 11.62 kB · 0 downloads

Terrain LOD is used to optimize object LOD (remove triangles that are under the ground). It happens with vanilla LOD or custom LOD.

The textures do not matching means just that. The textures defined by the full model do not match the textures with the same shape name in LOD model.
It typically happens because the full model was replaced. Otherwise the LOD model was replaced. Sometimes both if a mod does not know how to create LOD models properly.
The result can be that the LOD has wrong textures.

It seems the load order has issues without DynDOLOD output active. I suggest to fix those issues first.
If you believe assets from a mods are causing issues, then a troubleshooting step would be to disable that mod / hide those assets.

[w740su]

I generated LOD dozens of times in the past week. Now I fixed the mod list, I think I have found some more patterns:

First, I think it is related to the DynDOLOD output. It only happens often with the High DynDOLOD settings. I haven't met any performance issue without DynDOLOD or with low settings, and with medium settings there would be some rather short time after fast travel when frame rate drops, and I cannot tell if it is the same issue. I also tested trees and grass. Disabling ultra tree and grass LOD for the generation doesn't seem to affect the issue at all. When using the high settings, the frame rate drop only happens when I have some rocks/mountains in my view, and if I open console and wait there for 5-10 min (10-20 min with ENB on) doing nothing, the frame rate/GPU power consumption will turn completely back to normal. Like the screen shots in the previous post, disabling some small rocks will usually result in some performance boost. I tried to wait and see if I wait long enough every time the performance issue could be solved, it turns out that only 5~6 locations would return to normal performance, and if I travel to some other slow locations, some of the previously slow locations will slow down again.

I was suspecting the new mods I used for landscape (Majestic Mountains Complex Material + Majestic Landscapes + BDS3) since I was using the non-complex material version + Simplicity of Snow before with 0 performance issue. Removing Majestic Mountains Complex Material from the mod list does mostly eliminate the frame rate drops but having a mod list with only Majestic Mountains Complex Material + Majestic Landscapes doesn't have any problem (this time I didn't generate terrain LOD though).

I also tried to find if there is any hardware issue. Removing shader cache, reinstalling GPU driver with DDU, and setting GPU power to max performance all does nothing. When the performance drops, the VRAM usually sits at around 9-9.5/10GB of usage on the OSD. Although in task manager sometimes it shows over 11GB of combined GPU memory usage, but the issue still happens even when the GPU memory usage shows only 9/33.9GB. I compared the new textures to the old ones and they are the same resolution, so I guess the VRAM isn't the issue. I also tried the Intel monitor tool PresentMon which shows an interesting graph of CPU/GPU busy time: When the performance drops, the GPU busy time stays the same as the frame time (usually 100+ms/frame with ENB), while the CPU busy time would drop to less than 5ms. Another finding is when I was waiting for the performance to turning normal, the GPU power usually has a 2 step raise: if the issue was causing the GPU to consume only ~200W without ENB, sometimes after a while it will rise to ~270W, and if I turn on ENB right now it will immediately drop to a bit higher than 200W, and after some extra time it will rise to 300+W and the performance is normal again. Maybe there's something going on with the complex material textures? I couldn't see any SSD usage spikes in task manager though.

Sorry my test took a while and thanks for reading such a long result. If there's no fix for it, I guess I'll just turn down some settings or bear with the current performance.

[sheson]

No logs for the new load order and LOD generation were provided.

44 minutes ago, w740su said:

First, I think it is related to the DynDOLOD output.

Read https://dyndolod.info/Official-DynDOLOD-Support-Forum#Rudimentary-Troubleshooting to narrow down the problem to either static terrain, object and tree LOD or dynamic LOD. If you determined static LOD, it can be narrowed down further by hiding the meshes for each of the 3 types.

44 minutes ago, w740su said:

if I open console and wait there for 5-10 min (10-20 min with ENB on) doing nothing, the frame rate/GPU power consumption will turn completely back to normal.

So the performance issues have nothing to do full models or LOD as they do not change?

44 minutes ago, w740su said:

disabling some small rocks will usually result in some performance boost.

Does this mean you open console, click a full model and disable it? Disabling full models does not affect static object LOD.

44 minutes ago, w740su said:

Removing Majestic Mountains Complex Material from the mod list does mostly eliminate the frame rate drops but having a mod list with only Majestic Mountains Complex Material + Majestic Landscapes doesn't have any problem (this time I didn't generate terrain LOD though).

So, the performance problems had to do with the full models of this mod?

44 minutes ago, w740su said:

When the performance drops, the VRAM usually sits at around 9-9.5/10GB of usage on the OSD. 

What is the actual physical VRAM size?

What you are reporting seems to be all over the place. To me that hints resource issues like physical VRAM being exceeded. Or may be some kind of temperature throttling. Enable and check the papyrus logs for script issues.
I would start testing things the other way around, with a vanilla game (including vanilla INI settings for LOD etc.) and then add ENB and then mods in groups to see their effects on resource usage and performance and if things stay constant or not. Then generate the LOD with the settings that keep it under whatever the threshold/edge is. Be aware of 3rd party "LOD" models or rules using full models or not really well optimized models for LOD.

[w740su]

Scratch all the reporting I wrote below. I think I've found the issue. My VRAM is probably throttling. Despite my GPU is only at 77C maximum, when I am trying to record a video to show the problem I found even when the performance issue "fixed itself" after waiting for a while, if I keep waiting there longer the performance could drop agian. Then I have HWiNFO opened for monitoring all the temperature sensors and it turns out whenever I have mountains/rocks in sight the GPU memory junction temperature will break 100C(HWiNFO says 110C is the limit), and if I stop turning the camera, after a several minutes it drops to 90C, in sync with the framerate drops. Probably because some rock textures are 4k and 8k so only these objects will push the temperature to over 100C, and this happened to just the same time I changed my landscape mods. The GPU is kind of old so I'll order some thermal pads to see if this finally fixes it. 

 

Thank you so much for reading my lengthy posts these days.

 

Spoiler

Here is a list of what I've tested, the full mod list is the same I used when asking this question, and tested all these on new games.

Full mod list+DynDOLOD high=frame rate issue

Full mod list+DynDOLOD high-DynDOLOD output=no issue

Full mod list+DynDOLOD low=no issue

Full mod list+DynDOLOD high-ultra tree LOD-grass LOD=frame rate issue

Full mod list-DynDOLOD NG dll+DynDOLOD 1.6.353 dll& scripts+DynDOLOD high=frame rate issue

Full mod list+128/720p TexGen(256/1440p in other testings)+DynDOLOD high=frame rate issue

Full mod list-Majestic Mountains+DynDOLOD high=no issue

Only environment related mods+DynDOLOD high=frame rate issue

Only Majestic Mountains complex material+Majestic landscapes+DynDOLOD high=no issue

The rest time I was trying different DynDOLOD settings to see lowing which rule could fix. I hasn't found it yet.

9 hours ago, sheson said:

Does this mean you open console, click a full model and disable it?

Yes, here is a video to better show this, and it feels like all the rocks have some sort of issue, so when I reduce the lod settings there might be less rocks loaded. My current test routing is like this: Dawnstar -> Riverwood -> the Guardian Stones -> Pinewatch -> Falkreath -> Angi's Camp -> Half-Moon Mill ->Hunter's Rest -> Gjukar's Monument -> Rorikstead -> Old Horldan Karthwasten -> the Lover Stone -> Salvus Farm -> Haelgen -> Alchemist's Shack -> Ivarstead -> Snow-Shod Farm, and only in the underlined locations I have the issue. For locations like Half-Moon Mill, it was one of the heaviest part in my current mod list especially when facing west, but the performance issue only happens when I turn to the north east side facing the rocks by the road. 

9 hours ago, sheson said:

What is the actual physical VRAM size?

The physical VRAM size is 10GB and I was thinking the same that 10GB might be not enough, but with all on screen monitoring software I tried, the VRAM usage is at maximum ~9.5GB. The only place I could sometimes see more than 10GB of VRAM usage is in the task manager, but the issue still happens even when in task manager there's less than 10 GB in use. And I also compared some of the textures from my previous landscapes and they're the same resolution, the major difference is now there are complex material textures.

For throttling at least it's not caused by thermal as even at full power the GPU temperature is only 77C and when frame rate drops the GPU temperature drops to 60C.