GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[Kelmych]

A new simpler (and more robust) approach for handling uncompressed files is now used for the batch file and subsequent optimization for optimizing STEP mods and other non-vanilla textures. (the approach used for the vanilla textures is fine and didn't need revamping). These changes are reflected in the new version 2.7 of the batch files on the guide. The QuickStart guide and the Vanilla texture optimization sections have been updated for the new batch files. The "Optimizing the STEP Mods" section is still being edited for consistency with the new batch file.

 

This version changed only the batch file ("4_Mod_Sorting_Pre-optimization.bat") used to process STEP (and other) mods. It also didn't change the resulting optimized texture output for the representative mods optimized in the QuickStart tab when using the previous version, but it made the batch file much more reliable in handling arbitrary Skyrim mods. The previous batch file sometimes needed be be reconfigured to handle mods with uncompressed textures that hadn't previously been tested with it . The new batch file version and instructions uses DDSopt in a way that it handles all uncompressed textures properly even when they are mixed with compressed ones, and the batch file no longer needs some special configuration steps for uncompressed textures.

[Octopuss]

Oh what the...

I redid the whole optimization process this afternoon with the 2.65 batch file, and now you tell me this? Omg lol! :D

What are the main differences?

[Kelmych]

Nothing changed that would affect what you did.

 

The vanilla optimization didn't change at all from 2.65; it should have made more clear (I'll edit my post to do so). The only changes lately to the vanilla batch file was that an explicit path was added for "robocopy" to fix a problem and that a 2k uncompressed male body MSN is now moved directly to the output since it can't be reduced in size without corruption (as discussed in the troubleshooting forum), and these changes are included in 2.65. The batch files probably need an actual changelog vs. the running one that is posted here.

 

The major change is to the processing of STEP mods. The QuickStart now has a fairly short process for optimizing a set of STEP mods. I tested it yesterday and it worked well and takes much less time than the vanilla texture optimization since the vanilla textures are much larger than any other collection of mod textures such as the textures in section F and G of the STEP guide.

 

Sorry it wasn't clear in the initial post.

[DoubleYou]

Well, I finally took the plunge and optimized my vanilla textures the way the DDSopt guide says instead of Neo's version. It definitely was a lot more work, but I think it was well worth it, as it's going to save me a few gigs off Neo's method and seemed more professional. I'm compressing my Vanilla Optimized with 7-zip right now so I won't have to do that again (it says it's gonna take 9 hours? I hope not!).

[Kelmych]

Did you follow the instructions in the QuickStart guide or the main guide sections; the QuickStart guide is usually adequate and is (hopefully) much simpler? Once the files are extracted and the HRDLC cleaned those instruction shouldn't be too much more complicated than the ones in SR:LE (hopefully). Let us know what seemed complicated or unclear so we can look at improving it. I'm working on a process for sorting/optimization of the vanilla textures that is a simpler, but I haven't finished testing it. Extraction, cleaning, and archiving (if used) would remain the same.

 

The SR:LE guide doesn't seem to archive the files after optimization. I assume MO must not care whether its a folder or an archive. I like to have them archived so they take less space and so they are all together in one place.

 

If you have an SSD the archiving is a lot quicker. If you use the batch file archiver it takes a while, but once started you can just let it run and come back later when it finishes.

[Octopuss]

I noticed the very high quality settings are gone from the guide, and thus for compressed normal maps the suggested setings are now to resize to 1024x1024. What's the reason behind this change?

[Kelmych]

Most of the comments about the vanilla optimization portion of the guide lately have seem to be about how complicated it is to follow, and we're trying to simplify it. The only difference between the Very High quality and High quality settings was that High reduces the size of 5 2Kx2K Skyrim-textures.bsa compressed tangent space normal maps and 5 Dawnguard 2Kx2K compressed tangent space normal maps. These textures are for individual items vs. textures that are used on common items, so it didn't seem that useful to preserve these normal maps at such high resolution. The uncompressed 2Kx2K malebody_1_msn.dds texture, used on all male bodies, is preserved at full resolution. If useful we can simply add a sentence about using 2Kx2K settings for normal maps for those wanting to preserve these.

 

The Very High Quality recommended settings are still there for mod textures.

 

[Edit: a sentence about using higher limits for Vanilla Normal Maps was added to the instructions]

[Octopuss]

Oh ok. Guess everyone is free not to set lower resolution :P

 

As for the simplifying, I think the main problem was in the amount of text and overcomplicated instructions rather than the procedure itself. That has been significantly improved though.

[Kelmych]

All of the settings are suggestions, and any changes are usually based on feedback from users.

[EssArrBee]

Has anyone reported issues with the BSA extraction built into MO? I just ran the latest bat files and it said I was missing some HRDLC files. I'm thinking the MO extraction might have a few hiccups. I'll use BSAopt and check to see.

[Kelmych]

No. You can look at the batch file to see what it is looking for. If you are running the 1_HRDLC_Clean batch files then MO may have already removed some of the extraneous files; if that is the case these aren't necessarily really errors. Batch files are unfortunately very fussy, and some things they report as an error aren't really errors. If you want to determine whether the MO extraction is correct you can look at the file tree screenshots in the DDSopt guide that show how many files and folders result when you extract each of the vanilla BSAs.

 

 

---

An alternative approach to the sorting and optimization portion of Vanilla Texture Optimization was added as an Appendix in the DDSopt guide. This uses the DDSopt GUI, similar to the vanilla texture approach in SR:LE. The approach in the DDSopt guide has more steps since it does more selective optimization (see Q3 in the General FAQ for details of what the guide's vanilla optimization does). It is based on some recent discoveries I made about how the DDSopt GUI works. Some users may prefer this new approach; the batch files involve fewer steps when they work properly but batch files aren't as robust as we would desire (e.g., I expect the problem SRB is seeing above is that the batch file has no good way of determining when errors occur). The DDSopt GUI approach has its own issues; incorrect entries too often mean the user gets to start all over. I haven't compared the elapsed time for the two approaches.

 

The Output from both approaches is the same; if you have already optimized the vanilla textures you don't need to change anything.

[EssArrBee]

I actually think that the MO BSA extraction might have an issue, unless it somehow detects redundant files or errors and refrains from extracting them. I could not find anything like that in the MO documentation, so some MO person will have to confirm. After seeing how many files and folders were in my vanilla extracted folder after running the HRDLC Clean bat file, I was missing about 20K files and hundreds of folders.

 

EDIT: What compression level is safe to use with BSAopt to repack stuff? I thought using level 10 was to high.

[Octopuss]

I guess the best would be to drop Tannin a PM either here or on Nexus.

[Kelmych]

I actually think that the MO BSA extraction might have an issue, unless it somehow detects redundant files or errors and refrains from extracting them. I could not find anything like that in the MO documentation, so some MO person will have to confirm. After seeing how many files and folders were in my vanilla extracted folder after running the HRDLC Clean bat file, I was missing about 20K files and hundreds of folders.

 

I've never used MO to extract files from BSAs and then use them externally. I was actually hoping that there might be a way to use MO in a batch file (or just use the extracted files it already creates) to extract files from BSAs since BSAopt/DDSopt doesn't do this and none of the old TES4 tools for command line extraction work with TES5 BSAs.

 

There is a lot of redundancy in the Skyrim BSAs; many resources are duplicated causing significant BSA file size bloat.

 

EDIT: What compression level is safe to use with BSAopt to repack stuff? I thought using level 10 was to high.

I haven't personally tried doing this with the current Skyrim (1.9.32.0.8). I've read comments that even with reduced compression levels BSAopt-created BSAs with non-texture resources can cause CTDs. Optimizer Textures does repacking and seemed to be doing this successfully, but MotherGoose729 mentioned having problems lately with the BSAs it created. It's possible that Bethesda made some changes in BSA structure since version 1.6.89.0 (the last Skyrim version for which BSAopt BSAs didn't seem to have any problems) or that they began using some of the structure changes that had designed in earlier. Since Ethatron hasn't been available for quite a while I don't think anyone has looked at this. Perhaps Tannin might want/be able to look at the BSA header and structure; there is for example an old post on the BSAopt nexus page that suggests that BSAopt isn't using the latest changes in BSA header structure when it creates BSAs.

[DoubleYou]

I used Mod Organizer to extract everything but the HRDLC BSAs, and it worked fine. I used DDSopt to extract the HRDLC BSAs because I was multi-tasking, not because I thought Mod Organizer couldn't do it okay. I got no errors, so if there is an error somewhere, it would be in the HRDLC.