GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[Prophes0r]

Oh yes, those are correct. I keep seeing those 2 acronyms (about 12 times in this very thread) so I assumed they were a convention. As i said, i just got back into this a few days ago and I've basically been reading non stop.

 

As for the data, a combination of things.

In game I have the Elys MemInfo SKSE plugin, it is super light weight and works well.

I also was able to get Skyrim performance monitor working with SKSE but loaded by Mod Organizer

I also was grabbing data from sysmon on this machine and/or my second box.

and I turned on recording in a custom map of CPUz and GPUz to record the raw data for another run that I didn't include above just to confirm it.

 

I never got around to making them BSAs, I probably will when i get done with some other things.

I actualy started doing this to help me learn to deconflict stuff at a lower level.

 

I have 4 mods total in my "perfect" build, 2 that dislike each other, and 2 that crash each other. So I'm learning the creation kit now to see if i can remap some things, or combine them since there doesn't seem to be a reliable automatic tool that can handle it yet.

 

With all the mods i'd like to run, minus those 4 I'm clocking in at 2gb VRam/ 1Gb V-Vram / 2.4Gb Mem right now so if i can cut it down without bogging it would be nice.

[Kelmych]

A while ago I grabbed the text for the DDSopt guide and started to edit and expand it based on the posts on STEP and some of the forum posts referenced by Ethatron in the Nexus description. Partway through this process I found that I had some questions whose answers I wanted to include, but I didn't have enough information. I wanted to provide draft guide updates that reduced confusion vs. increasing it. If it would be useful I could continue the editing, if I had suggestions on the following:

 

1. There has been a lot of discussion on using DDSopt on textures beyond the vanilla Skyrim and HDDLC textures that obviously benefit from DDSopt processing. There are a few mod authors, e.g., Millenia and Cabal, whose textures are already recognized as being of sufficiently high quality that they shouldn't be run through DDSopt. There are some other mod authors who feel that their texture compression is already sufficiently optimized (there was an initial list in a STEP post recently). My approach for the latter group is to recommend running their textures through DDSopt and see if there are errors (as there were for many of the mods in the list) and if they are clean assume the mod doesn't need DDSopt.

 

2. Ethatron mentioned in this thread that body textures are not used in the same way as other textures, so these textures should not be run through DDSopt to improve the compression. It didn't sound like DDSopt recognizes these automatically, so I assume text needs to be added to DDSopt.ini so they are skipped (SKP). Does the texture file name need to include the path (e.g., actors\character\female\femalebody_1.dds) or is it sufficient to have only the texture name? Or should we just ignore the whole issue of special handling for body textures?

 

3. In the forum discussions referenced by Ethatron there was a lot of discussion on how to handle compression (if any) for normal maps. There are places in the DDSopt.ini file where text can be added to specify how to handle specific normal map textures, and the .ini file says that tangent and model space normal maps are handled differently in DDSopt than other texture files. Does DDSopt already recognize that a texture file whose name ends in _n.dds or _msn.dds are normal maps, in which case the .ini file entries are needed only for exceptions to the standard processing of normal maps? When I look at the DDSopt logs there are comments when it recognizes an alpha channel in a texture file, but there are no comments that indicate that DDSopt automatically recognizes textures as being normal maps.

[TechAngel85]

A while ago I grabbed the text for the DDSopt guide and started to edit and expand it based on the posts on STEP and some of the forum posts referenced by Ethatron in the Nexus description. Partway through this process I found that I had some questions whose answers I wanted to include, but I didn't have enough information. I wanted to provide draft guide updates that reduced confusion vs. increasing it. If it would be useful I could continue the editing, if I had suggestions on the following:

 

1. There has been a lot of discussion on using DDSopt on textures beyond the vanilla Skyrim and HDDLC textures that obviously benefit from DDSopt processing. There are a few mod authors, e.g., Millenia and Cabal, whose textures are already recognized as being of sufficiently high quality that they shouldn't be run through DDSopt. There are some other mod authors who feel that their texture compression is already sufficiently optimized (there was an initial list in a STEP post recently). My approach for the latter group is to recommend running their textures through DDSopt and see if there are errors (as there were for many of the mods in the list) and if they are clean assume the mod doesn't need DDSopt.

 

2. Ethatron mentioned in this thread that body textures are not used in the same way as other textures, so these textures should not be run through DDSopt to improve the compression. It didn't sound like DDSopt recognizes these automatically, so I assume text needs to be added to DDSopt.ini so they are skipped (SKP). Does the texture file name need to include the path (e.g., actors\character\female\femalebody_1.dds) or is it sufficient to have only the texture name? Or should we just ignore the whole issue of special handling for body textures?

 

3. In the forum discussions referenced by Ethatron there was a lot of discussion on how to handle compression (if any) for normal maps. There are places in the DDSopt.ini file where text can be added to specify how to handle specific normal map textures, and the .ini file says that tangent and model space normal maps are handled differently in DDSopt than other texture files. Does DDSopt already recognize that a texture file whose name ends in _n.dds or _msn.dds are normal maps, in which case the .ini file entries are needed only for exceptions to the standard processing of normal maps? When I look at the DDSopt logs there are comments when it recognizes an alpha channel in a texture file, but there are no comments that indicate that DDSopt automatically recognizes textures as being normal maps.

Please do! I just posted this over on another thread: https://forum.step-project.com/showthread.php?tid=1263&pid=18161#pid18161

 

That guide needs an overhaul. I've yet to use it do to confusion and it not being very noob friendly.

[Prophes0r]

It most definitely does try and compress the textures for all actors, including characters. I'm wondering if the INI can handle exclusions with wild cards so we can exempt the entire character (or even actor) directories.

 

I did try compressing Cabal's aMidianBorn Steel chest with the recommended settings, definitely ended up worse for wear even with just the base .dds

https://skyrim.nexusmods.com/mods/20792

 

 

Same for Hectrol's Dwarven Shield retex. Even just converting the main dds it comes out looking somewhat worse for wear.

https://skyrim.nexusmods.com/mods/3996

 

For the normal maps, if they are ALREADY lossy, and they have mips, shouldn't we just leave them? chopping them down but leaving them uncompressed causes them to end up bigger. Now if you could get the uncompressed masters, that is a different story. Same for lossy with no mips.

[Prophes0r]

Silly question. Where do we post images for the Wiki? and what format should they be in?

 

Ive done thousands of edits and creations on Wikipedia and Wikia, but i have never had cause to upload an image.

[TechAngel85]

Silly question. Where do we post images for the Wiki? and what format should they be in?

 

Ive done thousands of edits and creations on Wikipedia and Wikia, but i have never had cause to upload an image.

If you have editing privileges, you can upload a file under the Toolbox menu. Write in the summary box what the image is used for so it's not deleted later on.

[Farlo]

Silly question. Where do we post images for the Wiki? and what format should they be in?

 

Ive done thousands of edits and creations on Wikipedia and Wikia, but i have never had cause to upload an image.

As Techangel said, the Wiki isn't open for public editing yet since we've been working a lot on getting several key systems in place.  Hopefully in the near future we'll be opening the Wiki for editing by anyone with a forum account (the user databases are linked).  We're trying to keep the bandwidth down and since images take up a lot more than text we'd appreciate if you could host them elsewhere unless they're being used on a Wiki page.  I suggest installing Dropbox (use my referral and we both get extra space :D), you'll get a Dropbox folder when you install the app and anything you drop in the Public subfolder will be uploaded and can be linked to using the "Copy public URL" command in the context menu.

[TechAngel85]

Silly question. Where do we post images for the Wiki? and what format should they be in?

 

Ive done thousands of edits and creations on Wikipedia and Wikia, but i have never had cause to upload an image.

As Techangel said, the Wiki isn't open for public editing yet since we've been working a lot on getting several key systems in place.  Hopefully in the near future we'll be opening the Wiki for editing by anyone with a forum account (the user databases are linked).  We're trying to keep the bandwidth down and since images take up a lot more than text we'd appreciate if you could host them elsewhere unless they're being used on a Wiki page.  I suggest installing Dropbox (use my referral and we both get extra space :D), you'll get a Dropbox folder when you install the app and anything you drop in the Public subfolder will be uploaded and can be linked to using the "Copy public URL" command in the context menu.

 

I see what you did there....shameless. Haha! Imgur is another good place and is what I use for posting picutes/screenshots on the forums.

[Farlo]

 

Silly question. Where do we post images for the Wiki? and what format should they be in?

 

Ive done thousands of edits and creations on Wikipedia and Wikia, but i have never had cause to upload an image.

As Techangel said, the Wiki isn't open for public editing yet since we've been working a lot on getting several key systems in place.  Hopefully in the near future we'll be opening the Wiki for editing by anyone with a forum account (the user databases are linked).  We're trying to keep the bandwidth down and since images take up a lot more than text we'd appreciate if you could host them elsewhere unless they're being used on a Wiki page.  I suggest installing Dropbox (use my referral and we both get extra space :D), you'll get a Dropbox folder when you install the app and anything you drop in the Public subfolder will be uploaded and can be linked to using the "Copy public URL" command in the context menu.

I see what you did there....shameless. Haha! Imgur is another good place and is what I use for posting picutes/screenshots on the forums.

At least I pointed out that I was putting my referral in :P  Regardless of whether you use my link or not Dropbox is a very handy service for uploading and sharing files, otherwise I wouldn't be using it, much less recommending it.

[z929669]

A while ago I grabbed the text for the DDSopt guide and started to edit and expand it based on the posts on STEP and some of the forum posts referenced by Ethatron in the Nexus description. Partway through this process I found that I had some questions whose answers I wanted to include' date=' but I didn't have enough information. I wanted to provide draft guide updates that reduced confusion vs. increasing it. If it would be useful I could continue the editing, if I had suggestions on the following:[/quote']

If you have Wiki Edit auth, please feel free to begin editing, just as long as all info is correct to the best of our knowledge and suppositions are not stated as fact. There is still much to learn, and I am hoping Ethatron can provide more detail at some point.

1. There has been a lot of discussion on using DDSopt on textures beyond the vanilla Skyrim and HDDLC textures that obviously benefit from DDSopt processing. There are a few mod authors, e.g., Millenia and Cabal, whose textures are already recognized as being of sufficiently high quality that they shouldn't be run through DDSopt. There are some other mod authors who feel that their texture compression is already sufficiently optimized (there was an initial list in a STEP post recently). My approach for the latter group is to recommend running their textures through DDSopt and see if there are errors (as there were for many of the mods in the list) and if they are clean assume the mod doesn't need DDSopt.

I recommend the same. Most authors will say that their textures are already optimized. We ultimately would like to get a list of mods that benefit and those that don't play nicely (so that specific textures can be added to the INI)

2. Ethatron mentioned in this thread that body textures are not used in the same way as other textures, so these textures should not be run through DDSopt to improve the compression. It didn't sound like DDSopt recognizes these automatically, so I assume text needs to be added to DDSopt.ini so they are skipped (SKP). Does the texture file name need to include the path (e.g., actors\character\female\femalebody_1.dds) or is it sufficient to have only the texture name? Or should we just ignore the whole issue of special handling for body textures?

The INI requires the full path under textures ... RE not running body textures through DDSopt. I am pretty sure that the latest version can handle more than previous versions, so I would run all textures through DDSopt to see what results (as in #1 above). Then we are more certain of the INI additions.

3. In the forum discussions referenced by Ethatron there was a lot of discussion on how to handle compression (if any) for normal maps. There are places in the DDSopt.ini file where text can be added to specify how to handle specific normal map textures, and the .ini file says that tangent and model space normal maps are handled differently in DDSopt than other texture files. Does DDSopt already recognize that a texture file whose name ends in _n.dds or _msn.dds are normal maps, in which case the .ini file entries are needed only for exceptions to the standard processing of normal maps? When I look at the DDSopt logs there are comments when it recognizes an alpha channel in a texture file, but there are no comments that indicate that DDSopt automatically recognizes textures as being normal maps.

DDSopt does recognize the file-name conventions of *_n.dds, *_msn.dds, etc, so we only need to add exceptions to those conventions. I am still pretty unclear as to all of the uses of the INI and how it works

[Ethatron]

1. There has been a lot of discussion on using DDSopt on textures beyond the vanilla Skyrim and HDDLC textures that obviously benefit from DDSopt processing. There are a few mod authors' date=' e.g., Millenia and Cabal, whose textures are already recognized as being of sufficiently high quality that they shouldn't be run through DDSopt. There are some other mod authors who feel that their texture compression is already sufficiently optimized (there was an initial list in a STEP post recently). My approach for the latter group is to recommend running their textures through DDSopt and see if there are errors (as there were for many of the mods in the list) and if they are clean assume the mod doesn't need DDSopt.[/quote']

DDSopt can run in "repair"-mode, which was the initial motivation, and it can be used to create high quality DDSs from sources, which is the second motivation. Everything else is only to lessen my own suffering from badly authored textures (including Bethesda's mistakes), it's not meant to magically enhance textures, though lower mipmaps come out pretty much enhanced most the time.

Don't ask authors what to do, just do what you want. It's not about the search of absolute optimality.

2. Ethatron mentioned in this thread that body textures are not used in the same way as other textures' date=' so these textures should not be run through DDSopt to improve the compression. It didn't sound like DDSopt recognizes these automatically, so I assume text needs to be added to DDSopt.ini so they are skipped (SKP). Does the texture file name need to include the path (e.g., actors\character\female\femalebody_1.dds) or is it sufficient to have only the texture name? Or should we just ignore the whole issue of special handling for body textures?[/quote']

Some coloured textures are not actual colour-textures, like a few of the character textures are age-maps. If one treats them as colour-textures and filters them in sRGB-space it'll look a bit off (if you're sensible enough to these differences). DDSopt contains a classification database for all the textures, just look into the preview-information, it tells you exactly what type of texture you're looking at and what the channels contain. That was what I was refering to, you don't need to create global monstrous exception lists because I put generic "common knowledge" texture classes into the internal database, and age-maps went into it the moment I identified them as that.

3. In the forum discussions referenced by Ethatron there was a lot of discussion on how to handle compression (if any) for normal maps. There are places in the DDSopt.ini file where text can be added to specify how to handle specific normal map textures' date=' and the .ini file says that tangent and model space normal maps are handled differently in DDSopt than other texture files. Does DDSopt already recognize that a texture file whose name ends in _n.dds or _msn.dds are normal maps, in which case the .ini file entries are needed only for exceptions to the standard processing of normal maps? When I look at the DDSopt logs there are comments when it recognizes an alpha channel in a texture file, but there are no comments that indicate that DDSopt automatically recognizes textures as being normal maps.[/quote']

You should look at the preview-information if you want informations about a texture. The log doesn't contain messages of "identifications" of alpha-channels, it prints what is the statistics of the alpha-channel for the purpose of deciding if punch-through, explicit or interpolated alpha encodings can be used without loss of information. It is not related to the type of texture at all. Normal maps can also contain alpha-channels. The information is usefull if you convert uncompressed sources to DDS to decide if you go for DXT3 or DXT5 as target. Vurt's textures are a good example of mistaken good intent, he goes for DXT3, sadly in effect destroying quite some of the bark's alpha-channels.

[Kelmych]

Thanks for the clarifications

[Herrdoctor]

I'm seeing some visual glitches with the vanilla optimized textures. Smoke appears distorted and npc's suffer white pixelated blotches, particularly along the face. This occurs when ONLY the vanilla optimized textures are installed (as a compressed archive within Wrye Bash); no other textures are present, not even the optimized DLC packs.

 

I extracted, then optimized the bsa's (with the HD fixes) as per the instructions in this thread (and as general guidelines, the wiki) with 8.0 of DDSopt using the settings suggested in the earlier post linked to from the original post. I can only assume i've made a mistake. Or is this process somehow currently obsolete? I will upload some screenshots if this doesn't stand out as an obvious mistake on my part.

 

Also, I realize the step guide for DDSopt is in need of an update, but I doubt the guides are at fault for my texture distortions.

[Ethatron]

It's a beta, there can always be a bug (regardless the state anyway :^), if you can find the white pixelated blotches with the preview-tab I can see if I can reproduce it.

[z929669]

Ethatron has asked me to stage a Q&A on this thread. Please PM to me your questions about using DDSopt 8.0. Regarding your questions, please ...

• keep each question as brief, concise and specific as possible
• avoid asking multi-part questions. The answers we want will come, and we will even get unexpected answers to unasked questions.
• please don't ask questions about the DDSopt guide ... stick to DDSopt 8.0 interface, behaviors, results and methods only.

I will post each question prominently on this thread, and Ethatron will do his best to respond to each as promptly as time allows. Each Q&A 'block' will be posted on the OP and later used to build a FAQ that will be used by our site admin and Ethatron himself. Ultimately, this will help users better understand the program and Ethatron to better understand his users (at least from a STEP-user perspective).

 

Thanks

z