GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[stoppingby4now]

All well said' date=' but it still does not explain the reality of points made previously. Please speculate on the relative change in dynamic and dedicated VRAM in the performance charts of the DDSopt guide. As I implied, I think that these should be taken as conservative "just in case" numbers, hence a soft threshold, hence a safe bet that we can exceed the reported VRAM threshold by some amount, and 1.1x seems perfectly reasonable, given the data [/quote']

Clearing the wall of quotes.

 

Honestly, I don't see anything to consider as the data stands by itself, because we don't know what the data in the dynamic VRAM is, and if it is being swapped out (and based on those numbers, it is extremely unlikely you are experiencing any swapping at all). You can't correlate the increase in both as a directly mapped increase without knowing what is there, and it's purpose. Is it a remnant of another process that was swapped out, actual data used by the game, or driver related overhead?

 

I lean towards the latter, as I find it very difficult to believe it is directly related to the game, and more to do with the low overhead associated with the drivers (and in your case a dual GPU setup). The reported System GPU memory (or dynamic as reported by GPU-z) usage is far too low, and Dedicated VRAM usage is still below the threshold. I in turn ask you to consider if the drivers would stash any texture data to system RAM when there is still VRAM available? That wouldn't make sense at all, which is why I believe that has to be driver overhead.

 

Sorry, but I in no way buy the argument with that test data, and stand by the only real measure as being determining by frequency of swapping out of texture data (which will directly relate to actual use of texture data stashed away in RAM, and again, will have an impact on anything beyond 1.0 with increased frequency). 

---

EDIT: I did stumble upon the following response by a developer. The discussion is related to an nVidia card, but at least in terms of management as tied with Windows, those pieces should be the same regardless of manufacturer. I'm more than certain that what you are seeing reported for Dynamic VRAM (in relation to that test data) is driver overhead and nothing more.

A further question

How is the actual amount of system RAM allocated to each video card calculated ?

I tried several methods for this but none of them "compute" for me.

'ordinary' amounts of system RAM are allocated during initialisation for the regular data buffers & other support data like any other multibeam CPU task, and that's fairly small. Cuda runtime libraries, however, being proprietary & closed source, do things internally to support the detected hardware, 'gobbling' up some resources for fft libraries etc, some of which would be host side code, target card specific internal data structures somewhat dependant on Driver threads at a lower level, and various OS/driver level transfer caches & mirrored images. Some of those are part of WDDM specifications, some would be for device specific performance optimisation, and some of those would be Cuda specific to make the same Cuda code work on the different driver models, as part of an 'abstraction layer'.

[z929669]

Honestly, I don't see anything to consider as the data stands by itself, because we don't know what the data in the dynamic VRAM is, and if it is being swapped out (and based on those numbers, it is extremely unlikely you are experiencing any swapping at all). You can't correlate the increase in both as a directly mapped increase without knowing what is there, and it's purpose. Is it a remnant of another process that was swapped out, actual data used by the game, or driver related overhead?

 

I lean towards the latter, as I find it very difficult to believe it is directly related to the game, and more to do with the low overhead associated with the drivers (and in your case a dual GPU setup). The reported System GPU memory (or dynamic as reported by GPU-z) usage is far too low, and Dedicated VRAM usage is still below the threshold. I in turn ask you to consider if the drivers would stash any texture data to system RAM when there is still VRAM available? That wouldn't make sense at all, which is why I believe that has to be driver overhead.

 

Sorry, but I in no way buy the argument with that test data, and stand by the only real measure as being determining by frequency of swapping out of texture data (which will directly relate to actual use of texture data stashed away in RAM, and again, will have an impact on anything beyond 1.0 with increased frequency). 

---

EDIT: I did stumble upon the following response by a developer. The discussion is related to an nVidia card, but at least in terms of management as tied with Windows, those pieces should be the same regardless of manufacturer. I'm more than certain that what you are seeing reported for Dynamic VRAM (in relation to that test data) is driver overhead and nothing more.

 

Perhaps, but if it is only driver overhead, why does it jump up suddenly from around 50MB to >120MB at the small window of Dedicated = 959 - 968MB? If it were overhead, then I'd expect it to be pretty constant based on the number of active processes (which is absolutely consistent in all of those rows of data, albeit the very first, which is the windows background that should be subtracted from each of the other rows).

 

At this point, there really is no argument but that of the manner in which we speculate, as neither of us has provided any conclusive evidence (and it seems like it would be difficult to come up with any). We are left with speculating as to the underlying causes of the data we see, and I think that the jury is out. I am anxious to concede though, given a bit of light in this seeming black box ;)

[stoppingby4now]

Perhaps, but if it is only driver overhead, why does it jump up suddenly from around 50MB to >120MB at the small window of Dedicated = 959 - 968MB? If it were overhead, then I'd expect it to be pretty constant based on the number of active processes (which is absolutely consistent in all of those rows of data, albeit the very first, which is the windows background that should be subtracted from each of the other rows).

 

At this point, there really is no argument but that of the manner in which we speculate, as neither of us has provided any conclusive evidence (and it seems like it would be difficult to come up with any). We are left with speculating as to the underlying causes of the data we see, and I think that the jury is out. I am anxious to concede though, given a bit of light in this seeming black box ;)

In everything I've been able to dig up, there is driver overhead that will end up getting reported as use in dynamic VRAM. That alone means that a baseline needs to be taken to subtract from (hence your first row as a baseline). But, Process Explorer will be able to give you an accurate figure for just TESV.exe (which needs to be done, as we don't how accurate the figure is when taking the system wide usage as reported by GPU-z and subtracting the Windows baseline). Test on my system (using Process Explorer), my System GPU Memory for TESV.exe only fluctuated between 22.3 and 22.6 MB when switching between no SRO and SRO, but the global use fluctuated as much as 5MB.

 

Now, I admit, I didn't catch the two cases where the System GPU Memory (sorry, I don't like the term Dynamic VRAM) jumped over 100MB usage. I wholeheartedly agree that it is most likely texture data that was offloaded. But that doesn't change the fact that we need to know the frequency in which data is swapped out. But, looking again at your foot notes, you have c to designate that stuttering occurred at > 0.5x for Dynamic VRAM. Am I interpreting that accurately to mean that in those two scenarios you experienced stuttering (which would correlate to a higher frequency of swapping)?

[z929669]

Perhaps, but if it is only driver overhead, why does it jump up suddenly from around 50MB to >120MB at the small window of Dedicated = 959 - 968MB? If it were overhead, then I'd expect it to be pretty constant based on the number of active processes (which is absolutely consistent in all of those rows of data, albeit the very first, which is the windows background that should be subtracted from each of the other rows).

 

At this point, there really is no argument but that of the manner in which we speculate, as neither of us has provided any conclusive evidence (and it seems like it would be difficult to come up with any). We are left with speculating as to the underlying causes of the data we see, and I think that the jury is out. I am anxious to concede though, given a bit of light in this seeming black box ;)

In everything I've been able to dig up, there is driver overhead that will end up getting reported as use in dynamic VRAM. That alone means that a baseline needs to be taken to subtract from (hence your first row as a baseline). But, Process Explorer will be able to give you an accurate figure for just TESV.exe (which needs to be done, as we don't how accurate the figure is when taking the system wide usage as reported by GPU-z and subtracting the Windows baseline). Test on my system (using Process Explorer), my System GPU Memory for TESV.exe only fluctuated between 22.3 and 22.6 MB when switching between no SRO and SRO, but the global use fluctuated as much as 5MB.

 

Now, I admit, I didn't catch the two cases where the System GPU Memory (sorry, I don't like the term Dynamic VRAM) jumped over 100MB usage. I wholeheartedly agree that it is most likely texture data that was offloaded. But that doesn't change the fact that we need to know the frequency in which data is swapped out. But, looking again at your foot notes, you have c to designate that stuttering occurred at > 0.5x for Dynamic VRAM. Am I interpreting that accurately to mean that in those two scenarios you experienced stuttering (which would correlate to a higher frequency of swapping)?

Good question... I honestly forget now :(

[z929669]

Finally got around to doing some testing with the Pre5b version, and following are my image compares of foliage. In each image triplet, I have:

• original vanilla textures
• DDSopt-ed vanilla textures using large multipliers for:
  • Behave > Textures > Normal-map steepness raise
  • Behave > Textures >  Foliage-map opacity raise
  • Behave > Textures >  Color-map gamma
  • Behave > Textures >  Alpha-map contrast
• DDSopt-ed vanilla textures using SMALL multipliers for the aforementioned settings

Note that there is a bit of a tradeoff between certain textures. The larger multipliers (second in each set)correspond to better looking distant pine trees (see first two image sets), but unrealistically "lush" grasses (of certain types) and distant branches (see last image set). The opposite is true for the lower multipliers (third in each set).

 

I have not been able to identify the texture files involved with the branchy trees and yellowish grasses in the distance of the third image set, but those (and others like them) require special treatment I think (i.e., lower mip level foliage-maps should probably not be raised):

 

[Syrophir]

So what would be the latest recommended settings to optimize Skyrim - Textures.bsa?

[Besidilo]

So what would be the latest recommended settings to optimize Skyrim - Textures.bsa?

Second set in Z's comparison looks best, so use his recommended settings.

[Syrophir]

These https://forum.step-project.com/showthread.php?tid=228&pid=6366#pid6366?

Should I also use the default DDSoptPre5 Map multipliers from the Behave tab?

Last time ran with these settings the horizon turned purple, have to disable some files from the sky folder?

 

Thanks.

 

Edit: So far have ran DDSopt again with these settings on Vanilla Textures + HiResDLC and CombinedTexturePack 1.71.

 

Everything else is fine but have a purple horizon and some land is purple beyond the border. The horizon is fine if I skip vanilla sky folder, not yet tried to see which file it is. For the later I have no idea yet, maybe some mountain, rock or something.

[z929669]

Recommended settings for DDSopt

[Kelmych]

You mention SRO as an example of already optimized texture files. Is there some way we can tell from the texture files whether the textures have already been optimized well enough that DDSopt won't help, or do we need to create such a list for STEP based on previous STEP forum discussions and discussions with mod authors? By the way, I haven't seen almost any texture mod authors mention using texture optimization such as DDSopt in the mod description on Nexus or in comments posts.

[z929669]

You mention SRO as an example of already optimized texture files. Is there some way we can tell from the texture files whether the textures have already been optimized well enough that DDSopt won't help' date=' or do we need to create such a list for STEP based on previous STEP forum discussions and discussions with mod authors? By the way, I haven't seen almost any texture mod authors mention using texture optimization such as DDSopt in the mod description on Nexus or in comments posts.[/quote']

Mod authors that are familiar with standards of texture compression and layering do not need to use any specific tool, they only need to know the attributes of a given texture and the most efficient corresponding compression format (e.g., see Better Bedrolls and old version of Dragon Glyphs for examples of improper texture compression formats).

 

I am not certain what tools most mod authors use to manage texture compression, but I can understand why they might choose to leave that info out of their description (e.g., how many mod authors note the tools that they use, and if you note one of them, why not note them all? ... seems like unnecessary dev info that is irrelevant to most users).

 

An example of a good use for DDSopt (other than correcting some of the redundancies of specific texture packages) is what I recently did for myself: I isolated all of the *_n.dds normal maps of 2k SRO and ran them through DDSopt to constrain to 1024 and re-mip, then I replaced all of the originals (roughly 25% of all of those textures were affected normal maps). This effectively reduces the burden of the 2k package with a minimal quality loss ... 2048 color maps remain unchanged, but the size (and VRAM consumption) of the normal-map layers is reduced by one half (or as much as 25% in many cases).

 

Unfortunately, to determine if a texture pack is optimized, one pretty much needs to use DDSopt (or similar??) and examine the logs and results ... unless the author is known to be aware of best practices and meticulous enough to be trusted to have employed them.

[Kelmych]

You gave the answer I expected. If it were just a question of examining logs then it would be an straightforward task. Unfortunately, as you know, we need to look at the textures in context as you did in your recent analyses of the impact of optional DDSopt processing parameters. For the nearterm the only way I can think of to determine when to use DDSopt on textures in a mod is based on posts and discussions by the mod author and by other experienced texture creators/analysts that discuss the mod. Based on discussions in this thread, for example, there are certainly a handful of mod authors and associated mods whose textures we might recommend be left alone and not processed with DDSopt.

 

There aren't many 4K textures in the mods recommended by STEP. If I want to have 2K versions of these textures to reduce VRAM use should I use DDSopt on only the normal maps, as you mention above, along with the original 4K textures or should I reduce the textures themself to 2K using DDSopt?

 

Also, while I can certainly see the use of 2K textures for landscapes, if I want to get some additional reduction in Vram use would it make sense to use DDSopt to reduce some of the size of some item textures (e.g., jewelry) that are never displayed in a way that uses more than a small portion of the monitor?

[Syrophir]

Big thanks for the settings. ^^

 

Edit: SkyrimCloudsFill.dds is what makes the border horizon "line" between water and clouds purple/dark blue after optimization. Vanilla skies. :'(

[z929669]

Also, while I can certainly see the use of 2K textures for landscapes, if I want to get some additional reduction in Vram use would it make sense to use DDSopt to reduce some of the size of some item textures (e.g., jewelry) that are never displayed in a way that uses more than a small portion of the monitor?

 

It makes sense to manually reduce anything that will likely cost VRAM in outdoor areas using DDSopt if you have 1.5GB VRAM or less. Otherwise, you should be able to run most of the HR texture packs without worry (or the pre-built performance packs).

 

It is difficult to say for sure, but reducing clutter and misc items that are often in the world space will likely have a positive impact and would be worth it. I have decided to do this myself, but only after I have installed all mods so that I can do it all at once (rather tan by each package). Additionally, I have opted to reduce standard normal maps only as a first pass.

 

One DDSopt enhancement would be to allow mip stripping of only defined normal maps so that we can reduce all existing rather than only a single mip level at a time .... perhaps that functionality is possible now using the INI and specific settings, but I will have to confirm with Ethatron.

[z929669]

Big thanks for the settings. ^^

 

Edit: SkyrimCloudsFill.dds is what makes the border horizon "line" between water and clouds purple/dark blue after optimization. Vanilla skies. :'(

Thanks for the info. (wee need more like this )

 

I will verify when I have some time.