GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[Kelmych]

Wow, thank you for trying to epxlain it all to me. It is now pretty clear how to continue with DDSOpt! I actually followed Neo's instructions on DDSOpting the vanilla textures, which does not use the batch file. What is the different result to be expected from this approach?

The SR instructions use a single pass in DDSopt. The batch files support using reduced texture resolution for compressed normal maps, and reduced resolution and/or reduced quantization for uncompressed maps. This should reduce VRAM use relative to the SR instructions without much graphic quality loss.

 

And yes, the table is too complicated. I can give you some tips from the point of view of a noob/regular user (aka me) maybe it can help you to better structure it:

 

- As you mentioned yourself, the lack of recommendations means that the table has little guidance. After all, it is called a DDSOpt 'guide' :P 

That section was developed originally to help experienced users determine what recommendations we should be making.

 

- The column labels are rather incomprehensible for the average user that is looking for a few optimizations to reduce VRAM. What does it mean when it says no/yes in mip-maps? That the mod package is missing some mip-maps? Stuff like that could be explained at the beginning of the guide in a legend or in a seperate textbox. 

 

- Mip-maps, errors, compression and size could be combined in one column, or at least reduce the column width/labels to make space for new columns... read further below :)

Those 4 columns could be replaced by one that says whether DDSopt optimization (without reducing resolution)  is likely to reduce VRAM use (errors, compression, size) and/or GPU usage (mipmaps, since the GPU needs to generate these if they are missing). 

 

I think, but maybe I'm thinking too big, that it would be extremely helpful for the entire modding community if this became a reference table with information based on testing and/or reported user experiences: 

 

- A column labeled 'potential VRAM savings':The VRAM savings could be contributed by testing via the community. Perhaps open a new thread calling for people to test the difference between a DDSOpt and non-DDSOpt version of the mod. You could limit included results to 1680x1050 or 1080p resolutions. Attach a note saying that VRAM savings can vary greatly between systems. 

 

 - Another column labeled 'potential FPS savings'. Because not all users are interested in saving VRAM. Quite some people I know have issues with FPS instead of VRAM. For instance, DDSOpting Skyrim Flora Overhaul  (and I did not decrease resolution) reduced my VRAM use by only 20mb but it strangely enough also increased FPS in areas like the Rift by 4-6 with no noticeable quality loss. 

I'd like to understand why this happens; is it just eliminating some of texture features Vurt added that he mentions cause reduced FPS like some of his changes to grass?

 

- Another column labeled 'potential and/or known graphical complications from DDSOpt' (or a shorter title) could indicate problems that may/have been found to arise from DDSOpting the specific mod. E.g. DDSopting normals to half-size can sometimes increase shimmering. An extensive lsit of notes at the end of the table can explain texture 

The notes column was intended to allow including such issues when they are identified. If it doesn't work for such information we can try another way.

 

- Develop a benchmark standard. E.G.: 1) mod must be run in a vanilla environment with no other mods enabled 2) 'high' default settings must be used 3) ini must be default ini with no specific tweaks 4) the benchmark goes like: standing stones - riverwood - whiterun - fast travel to ivarstead - run to riften. Just some examples :)

 

 - Develop a DDSopt standard (e.g. exteriors to 1k textures/.5k normal and interiors to 2k textures/.5k normals) or develop multiple ones (e.g. low, medium & high res)

We've been working along those lines for a while. There were three resolution-related choices for vanilla textures, and we'll still need these. The textures were separated into sets of standard (color map) textures, compressed normal maps, and uncompressed maps. For STEP mods we will also need to have, if the user chooses it, twice as many sets so exterior textures can be separated from other textures. We've been trying to get the process of optimizing the vanilla textures established, and the results tested, lately; we're not quite done as there is another issue to resolve for this.

 

 

Of course, these columns will not be filled with data in a day. But this way, a database will gradually be built as users/step people contribute to it. Information can be extracted from the existing forum threads too, of course. The table should have a warning saying that the values are indications and/or estimates and therefor may vary widely per system. 

 

I already wrote down VRAM savings for every mod that I DDSOpted (about 10) so that could be a start, although I may have used less than optimal settings reading from your post of explanations. I realize that such a table can never be definite, or produce the same result for any user, but to have a roadsign is better than to be lost in the wilds :)

 

I'm just spitting out some ideas, that's all :P I'm willing to assist should you decide to embark on such a project.

Great! One example is testing and creating additional useful information on results of optimization (which you mention above); it seems to be the most tedious and time consuming part of the effort. 

[Guciec]

Sorry I saw my mistake in the interpreting of the process. And add a sentence to the guide to make it more dumb-proof: "On HDD drives coping process will take long, and executable window might look like frozen, select newly created folders and check properties, when its size reaches the size of the "Vanilla Extracted" directory the process is almost finished."

 

< Organizing the Vanilla Textures for Simpler Optimization ---> 'At the completion of this reorganization processing the Working Directory will contain a new temporary "Vanilla Optimized" directory and three additional temporary directories named "Vanilla Textures", "Vanilla Normal Maps", and "Vanilla Uncompressed Normal Maps". "Vanilla Textures" will have folders for each vanilla BSA, "Vanilla Uncompressed Normal Maps" will have one folder for each HRDLC, and "Vanilla Normal Maps" will have folders for the 4 vanilla Skyrim BSAs plus a folder for "Dawnguard" if it is used. None of these folders will be empty.'

 

I try to optimize Hi-Res DLC step by step but this fragment is causing me problems.

1) 'None of these folders will be empty.' sentence isn't explicit in saying which folders, the ones generally created in the Working Directory or the ones created in the sub-folder "Vanilla Normal Maps".

2) When I used 2_Pre-optimization.bat the effects weren't exact to the ones described in the quote above:

a) "Vanilla Normal Maps" has no folders inside, nothing actually

b) 'None of these folders will be empty.' whichever folders the author is thinking about there are empty folders.

 

Please, help and clarify the guide.>>

[phazer11]

Just a heads up to all those of you eagerly awaiting my video (yeah right) it'll be coming eventually. I can't say when exactly but it's getting there as Kelmych can probably attest. It all depends on when I get the time to sit down for another six-nine hours or so to make it. I'm currently working on my Digital Imaging and Digital Media (for websites) classwork. 

 

I can say that I have a fairly good video done already it has a few issues. One of the biggest being that it's 43 minutes long (that's still at least 20 minutes shorter than the first one)

[z929669]

Sorry I saw my mistake in the interpreting of the process. And add a sentence to the guide to make it more dumb-proof: "On HDD drives coping process will take long, and executable window might look like frozen, select newly created folders and check properties, when its size reaches the size of the "Vanilla Extracted" directory the process is almost finished."

 

Organizing the Vanilla Textures for Simpler Optimization ---> 'At the completion of this reorganization processing the Working Directory will contain a new temporary "Vanilla Optimized" directory and three additional temporary directories named "Vanilla Textures", "Vanilla Normal Maps", and "Vanilla Uncompressed Normal Maps". "Vanilla Textures" will have folders for each vanilla BSA, "Vanilla Uncompressed Normal Maps" will have one folder for each HRDLC, and "Vanilla Normal Maps" will have folders for the 4 vanilla Skyrim BSAs plus a folder for "Dawnguard" if it is used. None of these folders will be empty.'

 

I try to optimize Hi-Res DLC step by step but this fragment is causing me problems.

1) 'None of these folders will be empty.' sentence isn't explicit in saying which folders, the ones generally created in the Working Directory or the ones created in the sub-folder "Vanilla Normal Maps".

2) When I used 2_Pre-optimization.bat the effects weren't exact to the ones described in the quote above:

a) "Vanilla Normal Maps" has no folders inside, nothing actually

b) 'None of these folders will be empty.' whichever folders the author is thinking about there are empty folders.

 

Please, help and clarify the guide.>>

I revised the section for clarity.

 

@Kelmych

Please verify that I have depicted the proper folder structure in that section. I deleted my Working Directory.

[Guciec]

The guide in the section: Optimize the Vanilla Textures

Proposes to compress optimized files into the .7z achives, but why do it if you can fuse optimized textures in 'vanilla optimized' folder with 'Vanilla Extracted' files and repack files back into full BSAs, that in turn can be switched with the original ones. Why keep them in compressed files in wrye bash or MO? The installation of this packages will extract all files into loose files in the data folder wouldn't it? According to "BSAs and You" loose files increase the loading time because decompression from bsa archives is faster than using decompressed files.

[EssArrBee]

The guide in the section: Optimize the Vanilla Textures

Proposes to compress optimized files into the .7z achives, but why do it if you can fuse optimized textures in 'vanilla optimized' folder with 'Vanilla Extracted' files and repack files back into full BSAs, that in turn can be switched with the original ones. Why keep them in compressed files in wrye bash or MO? The installation of this packages will extract all files into loose files in the data folder wouldn't it? According to "BSAs and You" loose files increase the loading time because decompression from bsa archives is faster than using decompressed files.

They are the same speed on the user's end. Decompressing files from BSAs takes extra clock cycles, but not enough that a human can tell the difference while playing the game. We are talking about maybe a thousand cycles with 3 to 4 GHz processors, so the difference is .00000003 seconds or something even smaller. It's nanoseconds territory so just use whatever is easier for you.

[z929669]

The advantages of using loose files is that you have more control over conflicts, and you can see all conflicts. BSAs hide everything until runtime.

[Guciec]

But the article "BSAs and You" shows that while using HDD drive for skyrim *.bsa archives for vanilla bsas are better because reading time is noticeably shorter. And how vanilla BSAs can create conflicts if all the mods are loaded after them so have higher priority?

 

And isn't MO or Wrye Bash detecting and showing possible conflicts?

 

I understand unpacked BSA mods, but why vanilla BSA?

[EssArrBee]

The advantages of using loose files is that you have more control over conflicts' date=' and you can see all conflicts. BSAs hide everything until runtime.[/quote']

This makes sense for mods that come in BSAs, but not so much for the vanilla stuff. You wouldn't overwrite other mods with the unpacked BSAs of you vanilla assets.

 

Why can't someone port the FOMM bsa features to Skyrim. I have no problem opening Skyrim BSAs with FOMM to what's inside, so it can't be that hard.

[z929669]

The advantages of using loose files is that you have more control over conflicts' date=' and you can see all conflicts. BSAs hide everything until runtime.[/quote']

This makes sense for mods that come in BSAs' date=' but not so much for the vanilla stuff. You wouldn't overwrite other mods with the unpacked BSAs of you vanilla assets.

 

Why can't someone port the FOMM bsa features to Skyrim. I have no problem opening Skyrim BSAs with FOMM to what's inside, so it can't be that hard.[/quote']

I like to see what mod scripts, sounds and other assets are simple replacements over vanilla or are new content. I want to see the entire conflict tree.

[Kelmych]

Sorry I saw my mistake in the interpreting of the process. And add a sentence to the guide to make it more dumb-proof: "On HDD drives coping process will take long' date=' and executable window might look like frozen, select newly created folders and check properties, when its size reaches the size of the "Vanilla Extracted" directory the process is almost finished."

 

< Organizing the Vanilla Textures for Simpler Optimization ---> 'At the completion of this reorganization processing the Working Directory will contain a new temporary "Vanilla Optimized" directory and three additional temporary directories named "Vanilla Textures", "Vanilla Normal Maps", and "Vanilla Uncompressed Normal Maps". "Vanilla Textures" will have folders for each vanilla BSA, "Vanilla Uncompressed Normal Maps" will have one folder for each HRDLC, and "Vanilla Normal Maps" will have folders for the 4 vanilla Skyrim BSAs plus a folder for "Dawnguard" if it is used. None of these folders will be empty.'

 

I try to optimize Hi-Res DLC step by step but this fragment is causing me problems.

1) 'None of these folders will be empty.' sentence isn't explicit in saying which folders, the ones generally created in the Working Directory or the ones created in the sub-folder "Vanilla Normal Maps".

2) When I used 2_Pre-optimization.bat the effects weren't exact to the ones described in the quote above:

a) "Vanilla Normal Maps" has no folders inside, nothing actually

b) 'None of these folders will be empty.' whichever folders the author is thinking about there are empty folders.

 

Please, help and clarify the guide.>>[/quote']

I revised the section for clarity.

 

@Kelmych

Please verify that I have depicted the proper folder structure in that section. I deleted my Working Directory.

One small error that I corrected (STD in Vanilla Textures). (Your) Pictures are better than (my) words!

---

The guide in the section: Optimize the Vanilla Textures

Proposes to compress optimized files into the .7z achives' date=' but why do it if you can fuse optimized textures in 'vanilla optimized' folder with 'Vanilla Extracted' files and repack files back into full BSAs, that in turn can be switched with the original ones. Why keep them in compressed files in wrye bash or MO? The installation of this packages will extract all files into loose files in the data folder wouldn't it? According to "BSAs and You" loose files increase the loading time because decompression from bsa archives is faster than using decompressed files.[/quote']

In addition to what other responders have mentioned, for the DLC BSAs repacking the optimized textures with the other extracted DLC resources into a BSA can cause problems. A number of users have reported CTD errors after doing this. We don't really understand the cause. The default compression level used by DDSopt is higher than that used for the Bethesda files, but even if the DDSopt compression level is changed to the same level as Bethesda uses there are apparently still problems.

---

The advantages of using loose files is that you have more control over conflicts' date=' and you can see all conflicts. BSAs hide everything until runtime.[/quote']

This makes sense for mods that come in BSAs' date=' but not so much for the vanilla stuff. You wouldn't overwrite other mods with the unpacked BSAs of you vanilla assets.

 

Why can't someone port the FOMM bsa features to Skyrim. I have no problem opening Skyrim BSAs with FOMM to what's inside, so it can't be that hard.[/quote']

I don't understand what FOMM features you want ported. A number of utiltiies can open BSAs and show the contents (e.g., BSA Browser, BSAopt, DDSopt). Mod Organizer includes some of the FOMM features. Unfortunately Wrye Bash is not currently able to read the contents of a BSA, and WB is the only tool that shows conflicts among resources in installed mods. FOMM doesn't show conflicts across mods.

---

But the article "BSAs and You" shows that while using HDD drive for skyrim *.bsa archives for vanilla bsas are better because reading time is noticeably shorter.

As EssArrBee mentioned' date=' it isn't noticeably shorter, it's shorter but not appreciably so.

 

And how vanilla BSAs can create conflicts if all the mods are loaded after them so have higher priority?

They can't, of course, but you can have conflicts that you don't realize and hunting down conflict problems is very time consuming.

 

And isn't MO or Wrye Bash detecting and showing possible conflicts?

As mentioned above, only Wrye Bash displays conflicts and currently it doesn't detect conflicts with resources in BSAs. Mod Organizer handles, but doesn't provide a display of, conflicts including those in BSAs since it extracts BSAs into loose files when installing.

 

I understand unpacked BSA mods, but why vanilla BSA?[/quote']

[Kelmych]

 

I try to optimize Hi-Res DLC step by step but this fragment is causing me problems.

1) 'None of these folders will be empty.' sentence isn't explicit in saying which folders, the ones generally created in the Working Directory or the ones created in the sub-folder "Vanilla Normal Maps".

2) When I used 2_Pre-optimization.bat the effects weren't exact to the ones described in the quote above:

a) "Vanilla Normal Maps" has no folders inside, nothing actually

If you have the 3 HRDLC folders in Vanilla Extracted there should then be 3 folders in Vanilla Normal Maps. The batch file checks to make sure the individual HRDLC folders exist in Vanilla Extracted before copying to Vanilla Normal Maps.

b) 'None of these folders will be empty.' whichever folders the author is thinking about there are empty folders.

This should be true if all the vanilla BSAs were extracted to Vanilla Extracted.

 

Please, help and clarify the guide.>>

[Nearox]

z929669 and Kelmych: I'll try to add my findings to the wiki. I didn't know we could add them too (though I suppose that would be quite logical since it's a wiki... silly me :P).

[Kelmych]

I modified the first 4 columns in the STEP mod table in the guide. These are now:

• Did DDSopt improve the textures by noticeably reducing total texture size or adding missing mipmaps
• Is DDSopt recommended to improve textures for this mod (No means the author has already optimized it sufficiently, so use DDSopt only if needed to get a reduced resolution texture that isn't available in the mod).
• Does the mod include exterior textures; when VRAM reduction is needed these are usually the best candidates for size reduction, particularly the tangent space normal maps
• 4th column is TBD

The new first column combines the results of the original columns 1-4.

 

The intent of the table is that DDSopt should always be used if column 1 is Yes, it is recommended unless Column 2 is No (in which case it should be used carefully if used at all), and is often particularly useful for VRAM reduction for mods with Yes in column 3.

 

Some additional recommendations need to be added about handling uncompressed and compressed normal maps.

 

Let us know if there are disagreements about entries or columns in the revised table; this is the initial version.

[Kelmych]

Here are screen images for different versions of snow01_n.dds . The same Winterhold save file that z provided was used, but the images here are looking away from Winterhold. The view toward Winterhold has too much windblown snow in the foreground so it's hard to compare images. There is a noticeable difference between the images for vanilla and the optimized vanilla normal map textures, but no significant difference that I can see for the original/optimized SRO textures.

 

 

vanilla TSN            optimized vanilla TSN             SRO TSN                              optimized SRO TSN

 

Since there was a question about the first two images, I recaptured the screenshots in game and reposted them in this message