GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[Kelmych]

This is the introduction of new batch files for use in optimizing STEP mods. They are available here since they are still beta versions; I have tested them but they are not fully tested yet. The other batch files for use with the DDSopt guide are also included; it includes a small change in 2_Pre-optimization.bat for vanilla textures. The main new batch file is 4_Mod_Sorting&Pre-optimization.bat. The 5_Compress_Mods_7z.bat file is also included for automated archiving of the optimized mods.

 

The purpose of this batch file is to prepare one or more mods for optimization using DDSopt. The batch file sorts textures in the mod(s) into a set of temporary folders. Each temporary folder will typically be optimized with a different set of DDSopt Constraint tab parameters (format and resolution).  As with the batch files for vanilla textures, normal maps and uncompressed textures are separated from the rest of the textures. The batch file works with one or more mods, so it can be used incrementally as new mods/updates become available or you want to try a different strategies to control VRAM use. As with the previous batch files, the purpose is to reduce some of the tedious tasks involved in optimizing.

 

In addition, there is an option to separate exterior textures from other textures. Exterior textures typically cause greater VRAM use than other textures, and there has been much discussion in the DDSopt threads about the advantages of reducing the resolution of these textures to half that used for the rest of the textures (interior, NPC, etc.). This batch file supports two different ways of separating exterior textures.

 

For mods placed in the "Mod Extracted" directory, any textures in the terrain, landscape, and architecture directories are moved to either the "Mod Exterior Textures" or "Mod Exterior Normal Maps" temporary subdirectories.

 

For mods that have been released with multiple resolutions, a different approach can be used.  The higher resolution version of the mod is placed in "Higher Res Mod Files" subdirectory and the lower resolution version in "Lower Res Mod Files" subdirectory. The lower resolution version of the mod is used for exterior textures (textures in the three directories with exterior textures are moved to either "Mod Exterior Textures" or "Mod Exterior Normal Maps"). The higher resolution version is used for all other textures; textures in this version (excluding those in the three directories of exterior textures) are copied to the "Mod Ordinary Textures", "Mod Normal Maps", or "Mod Uncompressed Textures" temporary subdirectories. The higher resolution version is also used as the source for any non-texture files from the mod.

 

Uncompressed textures (excluding any exterior textures since there are almost never any uncompressed exterior textures) are also sorted automatically into a single directory. If you feel the uncompressed normal maps should be in a different folder that the other uncompressed textures let me know. Note that there are very few uncompressed textures in mods, and most of these are in body mod textures especially in skin-related textures.

 

Currently I included terrain, landscape, and architectural texture folders in the exterior category. If there are suggestions to include other textures, or to exclude portions of these, let me know.

 

One of the advantages of Optimizer Textures that has been mentioned is that it can optimize a large set of mods at once; unfortunately it is hard to be very selective in which mods and textures are optimized. With these batch files a large set of selected mods can be optimized using DDSopt without much time needed for user interaction.

[TechAngel85]

 

Good job there!

[phazer11]

Interesting to hear Kelmych I look forward to more polished versions. Hey who knows maybe they polished versions will coincide with December and I'll finally be able to finish the DDSOpt video. Interestingly I noticed something, if I record myself on my cellphone my voice is much less squeaky than through even my gaming headset (still missing a lot of the bass but hey!)

[Tiger]

I can't thank you people enough for all the helpful advice in this guide.

 

I installed full STEP a week ago. I used Alternate Start in Riverwood and did a test run through Bleak Falls Barrow. This was the result:

 

 

 

 

 

The most worrying thing was the memory peaking at 2.9Gb. I also had the common CTD in the wilds randomly.

 

I then spent the next few days carefully going over every mod. I cleaned everything in Tes5edit, extracted and optimized the official stuff. I then did the same for all STEP mods and used half sized normal maps where appropriate. I also got rid of a number of less essential mods, Birds, Ants, Footsteps etc etc.

 

Did another clean install and reapplied all remaining mods and did another test run:

 

 

 

 

Peak memory has been reduced by a huge amount and I can notice no real drop in quality. The game looks the best it has ever been for me. Admittedly with my initial install I chose a few mods at higher resolutions but after redoing everything I always took the lower resolution versions. With everything clean and optimized, I played for 8 hours solid last night without a stutter, crash or anything. I never imagined that the game could run so much better with the right care and attention.

 

So, thank you to everyone involved in this project. You've made me a happy Dovahkin.

[Spock]

Thank you so much Kelmych and all the others contributing to this guide!

The development of the batch file is really, really helpful. The approach I'm currently thinking about is this: Install everything, use your batch to shamelessly DDSopt my whole data files folder, then reinstall each mod which shouldn't be optimized (BAIN is such a nice, powerful tool to do that :)).

Maybe I can cut 95% of the time required this way. What do you think?

 

Regards,

 

Spock

[phazer11]

Please don't optimize your whole data files folder. Especially if you have mods! The batch files were meant to be used with the guide and require the vanilla textures to be extracted and then the batch files put them into the right folder structure, etc. Please follow the guide, trust me you'll appreciate going through the whole thing when you go on to optimizing your mods.

[Aiyen]

If you do it that way, then I can already now suggest my "wonderful" guide on how to identify missing textures, or mis-optimized textures etc. in the support section! :)

[Spock]

I'm sorry, I don't really understand the difference between those two approaches (if I don't want to re-size anything):

 

1. extract every single mod marked with "DDSopt recommended: yes"; use the batch file for mods to sort the textures; optimize according to guide; repack; install.

 

2. install everything; use the batch file for mods to sort all textures; optimize according to guide; overwrite datafiles; reinstall every mod marked with "DDSopt recommended: no" using BAIN (so the install order is maintained and the optimized textures get replaced with the originals).

 

From what I understand the outcome in the data files folder should be the same. I'm not sure I fully understood the guide to DDSoptimizing mods though...

[Aiyen]

I understood it as you wanted to optimize everything after you installed it... and that is a bad idea that will most likely ruin something!

 

Since I only do manual ddsopting, then I will leave this in the hands of the experts who have actually used the batch files!

[z929669]

If you do it that way' date=' then I can already now suggest my "wonderful" guide on how to identify missing textures, or mis-optimized textures etc. in the support section! :)[/quote']

You should put it as a mini guide or as a chapter to an existing guide on the wiki. Troublshooting Guide seems best, IMO. Links from other gudes where relevant, of course.

 

Then you can use a Guide thread to capture questions/comments.

[Kelmych]

The vanilla texture batch files have been used by many people by now and there don't seem to be any significant problems using them. They include some fixes (particularly removal of duplicate files from HRDLC and keeping several problematic vanilla textures from being optimized) that would otherwise be very tedious to accomplish. We certainly recommend these.

 

The new STEP mod batch files haven't had much testing yet. The batch commands are fairly general (unlike the vanilla batch files, there are no commands that specifically address a small set of files) and use fairly straightforward features of the Windows robocopy program, but it's always possible there could be occasional issues since robocopy (Robust Copy) has some limitations. So far the only issue I've seen in my testing is that mods with multiple optional resolutions in the mod file (e.g., the All-in-one installer for Real Ice) don't work properly with the current version of the batch files (the batch files function without error, but they don't separate out the optional resolutions so they can be optimized with different parameters). Most STEP users are doing the STEP mod optimizations manually, but I find this a little tedious (especially if you want to use different resolutions for exterior textures) and it's also fairly easy to make errors when using the filtering features of DDSopt. If you use the new batch files I suggest using it with a small set (perhaps ten mods or less) per run for a while until there has been more testing. Of course, let us know if you encounter any problems. I'm writing additional documentation for the new batch files and I'll provide this soon.

[Spock]

So far the only issue I've seen in my testing is that mods with multiple optional resolutions in the mod file (e.g., the All-in-one installer for Real Ice) don't work properly with the current version of the batch files (the batch files function without error, but they don't separate out the optional resolutions so they can be optimized with different parameters).

 

Sounds like it would actually be preferable doing it the way I suggested :D

 

Is the batch file separating the files significantly different from the way you'd do it by hand?

[Kelmych]

I'm sorry, I don't really understand the difference between those two approaches (if I don't want to re-size anything):

 

1. extract every single mod marked with "DDSopt recommended: yes"; use the batch file for mods to sort the textures; optimize according to guide; repack; install.

 

2. install everything; use the batch file for mods to sort all textures; optimize according to guide; overwrite datafiles; reinstall every mod marked with "DDSopt recommended: no" using BAIN (so the install order is maintained and the optimized textures get replaced with the originals).

 

From what I understand the outcome in the data files folder should be the same. I'm not sure I fully understood the guide to DDSoptimizing mods though...

#1 is the recommended approach, although I suggest optimizing a portion of the mods then doing another portion vs. one single huge optimization. The optimization process works on the textures in individual mod files, not on the textures installed in Skyrim\Data.

 

I don't really understand #2. You seem to be saying optimize all the texture files in Skyrim\Data\textures (which would take a very long time with DDSopt and optimizes many textures which shouldn't be optimized), then remove all the textures that shouldn't be optimized by reinstalling the original versions of the mods. Among other issues, this will cause significant problems with the vanilla textures which are separately optimized. It will also be tedious since Wrye Bash won't allow two versions of a mod with the same file name, and if the names are different I don't see how WB maintains the proper installation order. I doubt it would work well with Mod Organizer either. I also don't see how it can work in a maintenance mode when updates to mods, removal of mods, and new mods are periodically being added.

[Kelmych]

So far the only issue I've seen in my testing is that mods with multiple optional resolutions in the mod file (e.g.' date=' the All-in-one installer for Real Ice) don't work properly with the current version of the batch files (the batch files function without error' date=' but they don't separate out the optional resolutions so they can be optimized with different parameters).[/quote'']

Sounds like it would actually be preferable doing it the way I suggested :D

 

Is the batch file separating the files significantly different from the way you'd do it by hand?

The first comment was discussed in the previous post.

 

Manually optimizing an individual mod with DDSopt involves:

• setting processing parameters in the Constraint tab
• enter a filter criteria using the filter window (the center window)
• using the filter to select a portion of the textures
• processing the selected textures with DDSopt

then repeating 1-4 for each different subset of textures that are processed with different Constraints tab parameters (e.g., exterior ordinary textures, exterior normal maps, non-exterior ordinary textures, non-exterior normal maps, and uncompressed textures). With 5 different subsets of textures and 4 manual steps per texture subset it takes 20 manual steps to optimize an individual mod (worst case).

 

The batch files take care of steps 2 and 3 automatically, take care of all of the texture subset types automatically, and can be used to simultaneously optimize multiple mods. They do this by automatically separating the textures into separate subdirectories prior to any DDSopt optimization. Thus steps 2 and 3 above are done first for all the mods and texture types, and then the optimization is done. With the batch files, optimizing textures takes 2 steps per texture type (steps 1 and 4) so there are only 10 steps, and these steps are simpler than those used when manually optimizing. Moreover, since the batch file approach can handle multiple mods simultaneously, only 10 steps are need for a large set of mods.

 

The time to run the batch files is a small portion of the total optimization time; DDSopt optimization takes 80-90% of the time. The total time is similar with both the manual and batch methods, but with the batch files the user can leave the computer while the DDSopt optimization is running so the batch approach takes less user time.

[Spock]

2. install everything; use the batch file for mods to sort all textures; optimize according to guide; overwrite datafiles; reinstall every mod marked with "DDSopt recommended: no" using BAIN (so the install order is maintained and the optimized textures get replaced with the originals).

I don't really understand #2. You seem to be saying optimize all the texture files in Skyrim\Data\textures (which would take a very long time with DDSopt and optimizes many textures which shouldn't be optimized), then remove all the textures that shouldn't be optimized by reinstalling the original versions of the mods.

 

That's exactly what I proposed to do.

Among other issues, this will cause significant problems with the vanilla textures which are separately optimized.

I already optimized the separately so I could just reinstall them as well.

It will also be tedious since Wrye Bash won't allow two versions of a mod with the same file name, and if the names are different I don't see how WB maintains the proper installation order.

I'm not sure I'm getting what you are trying to say here. I'm not planning to manipulate anything in the bash installers folder. Just optimize, then copy the optimized textures and replace those in the datafiles folder. I doubt Wrye Bash checks for file sizes and datestamps of .dds files, so it shouldn't notice anything has changed.

I also don't see how it can work in a maintenance mode when updates to mods, removal of mods, and new mods are periodically being added.

Yes, the method would have to be repeated every time anything changes. It is totally unsuited if you plan to change your mods periodically. But I'm still hoping I can install baseline STEP, a lighting solution and Requiem and then just play the game. I spend too much time with modding the previous TES games and ended up never really playing them.