GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[rootsrat]

I will reply to this in the other thread. Please do not create off topic, as this is not relevant to this thread.

[hellanios]

The batch file for preparation of the textures doesn't seem to work. It creates some folders (some with subfolders) but with no files within.

[Kelmych]

The batch file for preparation of the textures doesn't seem to work. It creates some folders (some with subfolders) but with no files within.

Did the log.txt file show any errors? Did the batch file run for a long time or only briefly? The initial file copies are made to the STD folder in the Vanilla Optimized directory; were any files copied there? Roughly how large (Mb) is the Vanilla Extracted folder?

[hellanios]

The batch file for preparation of the textures doesn't seem to work. It creates some folders (some with subfolders) but with no files within.

Did the log.txt file show any errors? Did the batch file run for a long time or only briefly? The initial file copies are made to the STD folder in the Vanilla Optimized directory; were any files copied there? Roughly how large (Mb) is the Vanilla Extracted folder?

log file is completely blank. batch file ran for a few seconds then closed. batch file created 4 folders 1.vanilla normal maps 2.vanilla optmized>std 3.vanilla textures>hrdlc1,hrdlc2,hrdlc3 4.vanilla uncompressed normal maps>hrdlc1,hrdlc2>textures>terrain,hrdlc3.       no files were copied to any of these folders. vanilla extracted folder after hrdlc was repaired is ~12,6 GB (~12.903 MB). keep in mind that i have dawnguard and dragonborn dlcs. from these dlcs I only extracted the textures not all the source types.

[Kelmych]

It only uses the texture files from the DLC and doesn't care whether the rest of the DLC bsa stuff is there, and it doesn't care which DLC are there or not. I've been making other small changes and I'll upload a new version very shortly. I don't think it will solve your problem but we'll see. It assumes you have administrative privilege.

[Kelmych]

A new version of the batch file has been uploaded. The changes made were to support performing the optimization without the HRDLC.

 

I am adding a short notice to the guide for the batch processing. The batch processing uses the robocopy program which has been installed in Wndows by default since Windows Vista. It is available for Windows XP but it isn't installed by default. It is part of the free Microsoft® Windows® Server 2003 Resource Kit Tools (this url is for the english language version; it is also available for other languages). It replaced XCOPY which was not very robust, and added some useful new features. Alternatively, if Wndows XP is being used XXCOPY can be substituted for Robocopy; it is free for personal use and I'm fairly sure that it supports the same syntax.

 

hellanios, if you are using Windows XP this might explain the problem you reported.

[Kelmych]

A short discussion of the advantages and limitations of Optimizer Textures and DDSopt has been added to the General FAQ tab in the DDSopt guide.

[hellanios]

Unfortunately, the new batch file didn't work. The same thing happerns. I have Win7 64bit.

[gwoody79]

I am optimizing the larger landscape mods included in STEP, and the part that I am unsure about is what constraints to use for normal maps. I am going to use the "Standard Quality" constraint found in the DDSOpt guide for both textures and normals, but unsure if I should do the normals with the compressed or uncompressed setting found in the DDSOpt guide.  

[Kelmych]

I am optimizing the larger landscape mods included in STEP' date=' and the part that I am unsure about is what constraints to use for normal maps. I am going to use the "Standard Quality" constraint found in the DDSOpt guide for both textures and normals, but unsure if I should do the normals with the compressed or uncompressed setting found in the DDSOpt guide.  [/quote']

The quality settings should be useful for STEP mods as well as the vanilla textures. The uncompressed normal map settings are used only when the mod has uncompressed normal maps; the vanilla textures do have some of these. For the mods you want to process, look at the Step Mod Texture Processing section of the DDSopt guide and see which of the mods you want to optimize have Note 1 or Note 4 indicating that the mod contains uncompressed textures; there are not many that do. If you are doing mods that are not in the list, you can still check some of the normal maps in the mods to see whether they are uncompressed (it's fairly unusual). The new Uncompressed Normal Map settings are only for use with uncompressed normal maps where you want to keep the same resolution but change the format (e.g.,  from R8G8B8 to R5G6B5) to get a small (50%) size reduction. The Compressed Normal Map settings are the same ones that have been recommended for quite a while, namely half the resolution of the color maps. Only the Uncompressed Normal Map settings are actually new in the guide. We are trying to make the guidelines clearer, including providing settings screenshots in addition to (probably too much) text discussion.

 

By the way, the STEP mod list in the guide was updated today and is now current (until the STEP 2.3 list is released).

[Kelmych]

Unfortunately' date=' the new batch file didn't work. The same thing happerns. I have Win7 64bit.[/quote']

You might want to try running some of the commands manually in a command window. Open a command window (run cmd.exe). You should see the word Administrator at the top left of the window. Change the drive and directory to where the "Vanilla Extracted" directory is located. Run the command "robocopy "%cd%\Vanilla Extracted\STD" "%cd%\Vanilla Optimized\STD" *.png /s" without the outer parentheses (keep the inner ones). This simply copies all the .png files in textures\interface from the STD folder to the Vanilla Optimized folder. You can highlight the command on this message and copy it to clipboard, then right click in the command window to copy it there (so you don't need to retype it). If there is an error see if you can determine what isn't correct, or post the error message here. If it seems to run then check the Vanilla Optimized directory to see whether the files were actually copied. If you type in the command window

robocopy /?

 

you will get a list of all the valid switches for the command, by the way.

[gwoody79]

Thank you very much Kelmych for that thorough explanation, it certainly clears up any doubts and confusion I had.

[z929669]

 

 

The OP condenses most of the userful info in this thread as doe sthe DDSopt guide. I just finished a detailed optimization of all vanilla textures. I treated terrain normals and MSN as special cases. Attached are my logs (named according to treatment). I think that it makes good sense to constrain all vanilla normals to 1k and to ensure that the mip algorithms be applied for all using moderate settings. I omit the foliage mip algorithm for all but vanilla textures though. The gamma correction and contrast enhancements are key though.

It looks like the diffuse terrain didn't get remipped (5093 skipped textures), by the way. Did you manually select the different categories you used? When you finished how did you repackage the data into archive files? Are you planning to do the DLC textures also?

I processed the diffuse terrain along with all the rest in my final run, so they passed the verification I assume. I forgot the PNG though, so I'll need to copy those over as well as the interface textures that make sense. It may be a good idea to put anything like these in the DDSopt INI so that you don't need to treat them in your BATs.

 

I have yet to test everything and confirm the results I expect.

 

I ran as many runs as you see logs using filters applied manually. Just repackaged into 7z fast/non-solid. Still have not messed with the DLC other than HRDLC.

Part of the reason for the new batch file is to automatically take care of any direct copies like the .png files so it doesn't require a separate set of user steps. It also eliminates the need for special handling to eliminate any text and other non-texture files since they are never copied from the extracted BSAs to the folders that DDSopt itself uses; DDSopt runs on some temporary folders that contain only texture files and it doesn't run directly on the Vanilla Extracted folder.

 

Which interface textures should be copied over directly, or is it best to just put them in ddsopt.ini; I expect the latter is the better approach for these if they are .dds files?

I think that most of the interface directory does not need optimization if for no other reason than these textures appear only in the "mini-game" interfaces, where mip-mapping isn't necessary (no angular/scalar changes) and the loaded texture resources are very limited. I would consider adding everything in this folder to the INI (unless there is something that I am forgetting about).

The new batch file seems to work well in my (limited) testing. The DDSopt Optimization tab of the DDSopt guide has been modified to describe the use of the batch file, which resulted in eliminating the steps where the DDSopt GUI was used to select and deselect portions of the textures. Screenshots have also been added to this section to make it easier to determine the settings to use in DDSopt while optimizing different types of textures. The Quickstart tab was also modified for consistency with this approach.

 

One of the purposes of the simplification was to reduce the errors that were happening too frequently as users were following the at-times-intricate steps in the DDSopt guide. We would appreciate comments on whether the changes help, and what issues remain that we are capable of addressing (the just-put-the-BSAs-in-a-folder-and-push-one-button-and-poof-everything-is-optimized is still a bit out of reach).

 

Currently the optimized output creates one archive file for each of the vanilla BSA files. Is this what DDSopt users prefer? I noticed, for example, that z's example above combined all of the HRDLC into a single file vs. 3 separate ones. Do users prefer to have some or all of the optimized normal maps in different archive files or integrated with ordinary textures (as it is in the current DDSopt guide)? I've planning to provide a small batch file that uses 7zip to separately archive each of the optimized loose file folders, but I'd prefer to wait until we have a better idea of how we want to organize the output.

I had use your first "revamped" BAT release from last week, and it worked like a charm, so if you are adding functionality or efficiencies that aren't breaking anything you had set up previously, it should work fine. I had looked into the possibility of adding user-prompted input dialog on the command line to allow people to input custom source and destination directories. This would work well, but I stopped short of this for the time being. Still a possibility though.

 

When I ran the BAT, I changed the folder references to my custom directories, and it worked as expected.

Unfortunately, the new batch file didn't work. The same thing happerns. I have Win7 64bit.

 

I am 99% certain that the issue is in your folder names or locations and/or the directory location of the BATs. Understand that "%cd%\" refers to the current directory.

 

Also, be sure that whatever text editor you are using does not add/remove any of the line breaks or add any funky characters when copying from the Wiki (i.e., use Notepad++, TextPad, UltraEdit, etc and avoid Windows default text editors).

 

@Kelmych

We may want to upload the BAT files within an archive that will construct the proper directory structure when extracted. This would alleviate any copy/paste issues people could have.

[hellanios]

As Kelmych stated in a previous comment, the problem might be with administrative privileges. In your instructions for the hrdlc repair you say that we have to run the start batch file as administrators. When I ran the batch file as administrator by right clicking on the file and then choosing the ''run as administrator'' the file wouldn't execute properly. But when I simply left-clicked on the file it ran fine. For vanilla preparation batch file you don't instruct to run it as administrator. I tried both ''run as administrator'' and just left click the file options but neither would execute properly.

 

A note about ''run as administrator''': the file runs for a few seconds but it won't even create the folders.

[Kelmych]

z, you make several excellent points.

• I'll put an archive with all the batch files on Dropbox (they are really tiny), and I'll add the url to the guide.
• I'll leave the text version but add a comment not to use a document text editor.
• I'll add the interface directory to ddsopt.ini 

I'll also add a comment about making sure that the Working directory is not in a Program Files or Program Files (x86) directory. It's mentioned early in the guide, but not in the batch files.

 

I'll add a comment about using alternate directory names. I wanted the names to be consistent with the directory names used in the guide to avoid any confusion, and I expected that those who wanted to use these would be familiar enough with how to change them with an editor. There were a lot of comments from brand new STEP users that I interpreted to imply that we may have included too many choices in previous versions of the guide.

 

The version I uploaded yesterday had two additions

• For almost every copy operation it checks to see whether the related vanilla BSA folder in "Vanilla Extracted" is present. This allows the batch to be used incrementally. For example, if a new DLC is added the batch file can be run, and then DDSopt on each of the intermediate folders. The output folder in "Vanilla Optimized" can then be archived and added it to the game. This can now be done with any of vanilla BSAs.
• It now adds large (2Kx2K) tangent space normal maps from the STD and DLC to the "Vanilla Normal Maps" folder so they can be reduced in size when run through DDSopt. The rest of the DLC and STD compressed normal maps are handled like ordinary textures. There aren't very many, but there are more of these than I would have guessed.

If you want to use the version with these features make sure to get the one I put in the guide late yesterday (there were some bugs in the previous versions I uploaded yesterday), or wait an hour or so when I'll have the Dropbox url in the guide. hellanios, you should try the version from Dropbox once I get it setup.