GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[Ethatron]

Well, I started skimming with IrfanView, and the rule (alpha-channel == opacity maps) is pretty much followed ... except the textures are buggy. There are very very few non-opacity maps used. The dragons are examples, there the alpha-channel contains the disipation animation (when the dragons desintegrate).

I'm not so sure scrollworktrim is a special case anymore, maybe the poster just used the old version.

 

You really have to walk around and look what's going on I think. If you have the normal-map and the colour-map side by side (in the IrfanView thumbnail-browser) and you see there is something in the normal-map but the colour-map is black or only half filled, then you maybe got a case. I found this one: "dungeons\caves\cavebaseplants01.dds". Which means I look at all textures from the beginning to "texture/dungeons" and found nothing special.

Then follow the ice-caves, it's likely they are all special ...

 

BTW, if we don't do this cataloguing and I go back to the i-don't-care-about-opacity-pre5 then all these textures are becoming real low quality without probably noticing. Better it's black and we hunt them down, and log all the special texture-information for the modding community.

[Kelmych]

@Ethatron

 

Q.2: In the DDSopt output, there are several lines that make up the Final Report. Ex:

 

Final report "D:\Skyrim Mods\Mod_Working\DDSopt\out":

 processed files: 48

 modified textures: 27

 skipped textures: 0

 fixed textures: 0

 fixed files: 0

 broken files: 0 (0 without fixed ones)

 planar (1x1) textures: 0

 changed texture formats: 16

 i/o delta: -36413610 bytes

 tex delta: -21670756 bytes

 

The information contained in certain of these is pretty obvious; however, a couple of variables are somewhat unclear as to what can be gleaned (highlighted). Could you briefly describe these and provide an example or two of values that might raise and eyebrow versus those you would expect (e.g., a previously "optimized" texture set might yield a very different result than an "un-optimized" texture set or even a result from applying constraints). I have seen very large positive bytes for i/o as an example.

A.2: That's tough. Let's try:

• "fixed textures": The number of corrupt textures that have been fixed, this are currently textures which are too short (few bytes missing at the end). I found one or two of those, I don't know which software creates these truncated textures.
• "fixed files": A remnant of NIFopt, will always be 0. Refers to repaired models mostly.
• "planar (1x1) textures": The number of textures which have only one distinct value. These textures may be converted to 1x1 size without loss. Though seems not all hardware likes that. Haven't yet understood what's the real cause.
• "i/o delta": This is the global incoming to outgoing delta. Say you process a BSA, then it takes the size of a file inside the BSA (possibly compressed, then it takes the compressed size), and does the processing and writes it into another BSA. The sum of those differences is that value. If a DDS is not touched it still is possible to get i/o deltas != 0 because the zipper in BSAopt/DDSopt is about 10% better than Bethesda's. If you decompress a compressed BSA to directories, this delta is likely positive (out is bigger than in).
• "tex delta": This is only the recorded change in size from changes to texture-formats, resolution and mip-maps.

What exactly do you want to understand? The DXT artifacts (DXT3 vs. DXT5)? Why normal maps in DXT sucks?

Some texture mod authors (e.g., Isoku) provide uncompressed tangent space normal maps; this is the case for all of Isoku's mods I've seen. Are uncompressed normal maps preferable to the compressed ones that DDSopt creates?

 

One example of truncated textures is in DSI High . DDSopt reports all of the normal maps as being truncated. Interestingly, this is the only Isoku mod for which DDSopt reports truncated textures.  There are a few other mods that DDSopt reports as having truncated textures (in addition to Soul Gems Differ and DSI High) ; if you want a list of these let me know.

[z929669]

Well, I started skimming with IrfanView, and the rule (alpha-channel == opacity maps) is pretty much followed ... except the textures are buggy. There are very very few non-opacity maps used. The dragons are examples, there the alpha-channel contains the disipation animation (when the dragons desintegrate).

I'm not so sure scrollworktrim is a special case anymore, maybe the poster just used the old version.

 

You really have to walk around and look what's going on I think. If you have the normal-map and the colour-map side by side (in the IrfanView thumbnail-browser) and you see there is something in the normal-map but the colour-map is black or only half filled, then you maybe got a case. I found this one: "dungeons\caves\cavebaseplants01.dds". Which means I look at all textures from the beginning to "texture/dungeons" and found nothing special.

Then follow the ice-caves, it's likely they are all special ...

 

BTW, if we don't do this cataloguing and I go back to the i-don't-care-about-opacity-pre5 then all these textures are becoming real low quality without probably noticing. Better it's black and we hunt them down, and log all the special texture-information for the modding community.

 

OK, If I am understanding you correctly, you are recommending that the most recent DDSopt v8.0 pre-v1.0 on the Nexus is the preferred implementation and that the community users should examine the results and note all of the oddballs so that they can be added to exceptions list of DDSopt (and the "needs-texture-replacers" list for the modding community)?

 

In which case, we SHOULD be using the latest release you have posted on the Nexus and reporting our findings here as per usual :P

 

so it is written, so it shall be done ...

[Ethatron]

This is good about normal-maps:

https://www.3dkingdoms.com/tutorial.htm

 

While experimenting I detected that the _msns are apparently not regular (or not purely generated) object space normal-maps. If you have the Photoshop plugin of nVidia you can normalize the map and it looks very different. I suspect a smart artist played with contrast to somewhat weird effect, god can't they just be high profile and use zBrush ...? I don't think it's actually mathematically correct (like the 4444 normals that DDSopt produces).

 

Some texture mod authors (e.g.' date=' Isoku) provide uncompressed tangent space normal maps; this is the case for all of Isoku's mods I've seen. Are uncompressed normal maps preferable to the compressed ones that DDSopt creates?[/quote']

UnDXTed normal-maps are preferrable, yes. If you have enough noise (no smooth regions) you can get away with 565 and 4444 quantized normal-maps. I use mostly 565 for MSNs and it works okayish, same for the HQ water textures.

Tangent-space normal-maps vary to much to say a yes or no. Often it's possible to go without much damage, often the uncompressed size is prohibitive. In reality, if someones wants to make great big nms, you do it half resolution and then use 565, that matches exactly the size of a DXT1. If an alpha-channel is there, it'd be 4444, and that may go bad. Depends.

 

One example of truncated textures is in DSI High . DDSopt reports all of the normal maps as being truncated. Interestingly' date=' this is the only Isoku mod for which DDSopt reports truncated textures. There are a few other mods that DDSopt reports as having truncated textures (in addition to Soul Gems Differ and DSI High) ; if you want a list of these let me know.[/quote']

In reality DDSopt itself doesn't do anything, the DirectX-driver loads those textures and saves them. In this turnaround the textures are "fixed" inside DirectX. I just drop the message when the w*h*pixelsize != filesize, because it should. Maybe they have some fishy alignment or something like that (DirectX will align data according to the "rules"). If you don't use DirectX you're free to write whatever, GIMP probably doesn't. Or it writes the resolution to some wrong value, no idea. It may be off without knowing.

No need to report.

OK' date=' If I am understanding you correctly, you are recommending that the most recent DDSopt v8.0 pre-v1.0 on the Nexus is the preferred implementation and that the community users should examine the results and note all of the oddballs so that they can be added to exceptions list of DDSopt (and the "needs-texture-replacers" list for the modding community)?

 

In which case, we SHOULD be using the latest release you have posted on the Nexus and reporting our findings here as per usual :P

 

so it is written, so it shall be done ...[/quote']

Well, I just think buggy textures should be noted. Someone's turning the transparency-shader on in a NIF and pong (against intent) texture disappears. Isn't nice.

Also, let's not blow it out of proportion, it's ~10 textures out of 18000 currently. It's not going to be that much more.

[z929669]

@Ethatron

---

Q5: Does the INI file only let the user declare compression overrides and exclusions (i.e., by listing full texture path/file), or are the section names meaningful in a programmatic sense (i.e., are textures listed under different sections treated differently by DDSopt)?

[Ethatron]

@Ethatron

---

Q.5: Does the INI file only let the user declare compression overrides and exclusions (i.e.' date=' by listing full texture path/file), or are the section names meaningful in a programmatic sense (i.e., are textures listed under different sections treated differently by DDSopt)?[/b']

 

A.5: The .ini's purpose is two-fold. Defining the compression on a joint texture-type (3 channels can mean 3 independent channels, or colour or a vector, or xy + height etc.). And defining the content of the alpha-channel. As there is no 4 channel joint data-format in use in games that I know (CMYK would be one fe., even though even there it's arguable if K is really in dependency with CMY). So a texture is always "joint"+"extra", "joint" is always present, that's the seed-plane, and "alpha" is optional. You define compression on the seed. This are the supported seeds:

; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be tangent-space normal-maps, enables specific compression
;   taking unit-halfsphere property into account
; - [b]compression-format is supported here[/b]
[NormalsTangentSpace]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be model-space normal-maps, enables specific compression
;   taking unit-sphere property into account
; - [b]compression-format is supported here[/b]
[NormalsModelSpace]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be color converted
; - [b]compression-format is supported here[/b]
[DiffuseColor]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be greyscale converted
; - [b]compression-format is supported here[/b]
[DiffuseGrey]

The supported extras are:

; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with opacity, enables specific
;   compression taking this into account
; - [b]compression-format isn't supported here[/b]
[AlphaOpacity]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with foilage, enables specific
;   compression taking this into account
; - [b]compression-format isn't supported here[/b]
[AlphaFoilage]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with something custom, disables
;   specific compression only working for opacity
; - [b]compression-format isn't supported here[/b]
[AlphaCustom]

All of them (seed + extra) have their own processing. Normals are normalized, tangent-space vectors are possibly stored partially derivative, opacity gives importance to colour (the more transparent the less important the exact colour-value), foilage is possibly made sharper. The settings (in the menu) correspond exactly to each of these types.

[z929669]

@Ethatron

---

Q.5: Does the INI file only let the user declare compression overrides and exclusions (i.e.' date=' by listing full texture path/file), or are the section names meaningful in a programmatic sense (i.e., are textures listed under different sections treated differently by DDSopt)?[/b']

A.5: The .ini's purpose is two-fold. Defining the compression on a joint texture-type (3 channels can mean 3 independent channels, or colour or a vector, or xy + height etc.). And defining the content of the alpha-channel. As there is no 4 channel joint data-format in use in games that I know (CMYK would be one fe., even though even there it's arguable if K is really in dependency with CMY). So a texture is always "joint"+"extra", "joint" is always present, that's the seed-plane, and "alpha" is optional. You define compression on the seed. This are the supported seeds:

; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be tangent-space normal-maps, enables specific compression
;   taking unit-halfsphere property into account
; - [b]compression-format is supported here[/b]
[NormalsTangentSpace]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be model-space normal-maps, enables specific compression
;   taking unit-sphere property into account
; - [b]compression-format is supported here[/b]
[NormalsModelSpace]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be color converted
; - [b]compression-format is supported here[/b]
[DiffuseColor]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to be greyscale converted
; - [b]compression-format is supported here[/b]
[DiffuseGrey]

 

The supported extras are:

; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with opacity, enables specific
;   compression taking this into account
; - [b]compression-format isn't supported here[/b]
[AlphaOpacity]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with foilage, enables specific
;   compression taking this into account
; - [b]compression-format isn't supported here[/b]
[AlphaFoilage]
; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; - mark textures to have an alpha-channel with something custom, disables
;   specific compression only working for opacity
; - [b]compression-format isn't supported here[/b]
[AlphaCustom]

 

All of them (seed + extra) have their own processing. Normals are normalized, tangent-space vectors are possibly stored partially derivative, opacity gives importance to colour (the more transparent the less important the exact colour-value), foilage is possibly made sharper. The settings (in the menu) correspond exactly to each of these types.

I think that I understand the terminology you are using ...

• joint texture = multiple channels containing diff info?
• seed-plane = "joint" = main channel?

Under the [behave] menu, there are five different main settings and four modifier settings.

• Are these what correspond to the "supported seeds & extras" you point out above?
• Is the INI simply allowing one to define specific textures as receiving one of those "Behave" treatments?
• Is this necessary when the file name does not properly confer the "joint" and "extras" types?

Point me to another resource if this is too much to answer conveniently ;) (I am still in the process of reading your other refs, so if they contain these clues, let me know)

[Ethatron]

You know something about colour-spaces? And/or about specific game data stuffed into textures?

Colour mostly consist of multiple dimensions (RGB -> wavelength samples, red axis, green axis, blue axis), game data very rarely has multiple dimensions, normals are 3D-vectors.

You can't rip RGB or XYZ appart, that's why I call them "joint" for you, because they are inseparable.

More later ... :P

[z929669]

When I run DDSOpt there are sometimes no compression changes (e.g.' date=' DXT5 to DXT1)' date=' no changed texture formats, and no tex delta, but DDSopt says there are modified textures. When this happens, is DDSopt still making changes to the texture data but the changes aren't affecting the size of the texture? One example of such a texture file is the Chests_Rich_Dirty mod in STEP.[/quote'']

Have you looked at the log file in detail? My guess is that there are changes not affecting size, but I would need to run this to convince myself that this message isn't always delivered after processing.

Yes. I chose the example file because the log file is simple. There are 9 textures, and the log says 9 processed files and 8 modified textures. The only notes, on 3 of the textures, say "Alpha-channel has 1 distinct value(s), without black or white." The log does say "i/o delta: -1362 bytes" which might mean that DDSopt recompressed the textures in the .rar file using a more efficient rar compressor. This isn't the only example. You might look at your log files and see if you can match the "modified textures" number with the comments. I can usually match it with some of the textures but not all. For this example I can't match the "modified" value with any textures.

 

Basically I don't know what "modified textures" means, and whether DDSopt changes textures even when there is no change in the texture size.

Posting this for Ethatron's comment.

---

You know something about colour-spaces? And/or about specific game data stuffed into textures?

Colour mostly consist of multiple dimensions (RGB -> wavelength samples' date=' red axis, green axis, blue axis), game data very rarely has multiple dimensions, normals are 3D-vectors.

You can't rip RGB or XYZ appart, that's why I call them "joint" for you, because they are inseparable.

More later ... :P[/quote']

---

"Vurt's Skyrim Flora Overhaul"' date=' well ... he destroyed a few of his textures with DXT3 compression:

- architecture\whiterun\wrtreealivebark01_n.dds (should be DXT5)

- architecture\whiterun\wrtreedeadbark01_n.dds (could be DXT5, should check PSNR in the compressonator)

- landscape\grass\forestgrass01_n.dds (could be DXT5, should check PSNR in the compressonator)

- landscape\grass\forestgrass02_n.dds (should be DXT5)

- landscape\trees\treeaspenbarkcomp_N.dds (should be DXT5)

- landscape\trees\treeaspenbranchcomp_n.dds (could be DXT5, should check PSNR in the compressonator)

- landscape\trees\treepineforestbarkcomp.dds (should be DXT1, is DXT3 black)

- landscape\trees\treepineforestbarkcomp_n.dds (could be DXT5, should check PSNR in the compressonator)

- landscape\trees\treepineforestbarkcompsnow_N.dds (should be DXT5)

- ... and so on, now I'm tired, actually all normal-map alpha channels should be revisited and probably almost all have higher quality as DXT5

 

I'm not sure if the bark-shader employs transparency, if yes, then all barks have black/transparent speckles.

 

If someones convinces Vurt to give out the uncompressed maps, we can make a proper compressed version for him (using the format with the least error).[/quote']

Vurt responded to this critique and provides explanation on this post.

[Ethatron]

When I run DDSOpt there are sometimes no compression changes (e.g.' date=' DXT5 to DXT1), no changed texture formats, and no tex delta, but DDSopt says there are modified textures. When this happens, is DDSopt still making changes to the texture data but the changes aren't affecting the size of the texture? One example of such a texture file is the Chests_Rich_Dirty mod in STEP.[/quote']

 

Recalculating mip-maps counts as modification. If nothing else happens the file-size is the same.

[TechAngel85]

I'm getting ready to DDSopt my vanilla textures. Which version of DDSopt is currently recommended? Also, I've never used it on the Hi-res DLC BSA files before, do I still need the ESP files once they're unpacked and optimized as I'm not planning to repack them and only use the loose files.

[z929669]

I'm getting ready to DDSopt my vanilla textures. Which version of DDSopt is currently recommended? Also' date=' I've never used it on the Hi-res DLC BSA files before, do I still need the ESP files once they're unpacked and optimized as I'm not planning to repack them and only use the loose files.[/quote']

You do not need the ESPs in this case, as they only exist to load the BSAs (not a good practice normally to eliminate plugins from the load order, but this is one exception). I keep everything so that Steam does not hog any of my bandwidth syncing (unless they finally fixed the forced-update ussue). Just keep the ESPs unticked and use the *.7z compressed folders to load using Wrye Bash. Suggest non-block compression for faster extraction when installing/uninstalling mods.

[TechAngel85]

I'm getting ready to DDSopt my vanilla textures. Which version of DDSopt is currently recommended? Also' date=' I've never used it on the Hi-res DLC BSA files before' date=' do I still need the ESP files once they're unpacked and optimized as I'm not planning to repack them and only use the loose files.[/quote'']

You do not need the ESPs in this case, as they only exist to load the BSAs (not a good practice normally to eliminate plugins from the load order, but this is one exception). I keep everything so that Steam does not hog any of my bandwidth syncing (unless they finally fixed the forced-update ussue). Just keep the ESPs unticked and use the *.7z compressed folders to load using Wrye Bash. Suggest non-block compression for faster extraction when installing/uninstalling mods.

Thanks! So is the current beta version recommended for use? Right now have the stable DDSopt 0.7.3 downloaded.

[z929669]

v8.0 revision 3 is best (second main file)

[Kelmych]

When I run DDSOpt there are sometimes no compression changes (e.g.' date=' DXT5 to DXT1)' date=' no changed texture formats, and no tex delta, but DDSopt says there are modified textures. When this happens, is DDSopt still making changes to the texture data but the changes aren't affecting the size of the texture? One example of such a texture file is the Chests_Rich_Dirty mod in STEP.[/quote'']

 

Recalculating mip-maps counts as modification. If nothing else happens the file-size is the same.

So, if I understand correctly, DDSopt can recompute and replace the mip-maps, which could be done without changing the texture file size since the mip-maps have fixed sizes. The mip-maps would be different than before but still the same size. Ideally they would be better than what was previously in the texture files. That makes sense, I just didn't realize DDSopt does this.