GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[alt3rn1ty]

Well I did mention that loose files always win, so if all your textures are loose files, they will be in game ( or at least the last ones you installed which overwrote the previously installed replacers files with the same name ).

 

The advantage to repacking in BSAs is the game loads them quicker from a highly compressed BSA. Ethatron mentions tests he did in the NIFOpt documentation here ( scroll way down near the bottom to the paragraph called 'A little Calculation' )

I know its not mentioned in the DDSOpt or BSAOpt documentation, but Ethatron is using the same BSA compression technology for all his utilities, so it got mentioned there as part of NIFOpts experiments.

A little calculation

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

Harddisk-throughput: 20MB/s

Memory-throughput: 2000MB/s

zlib-throughput: 80MB/s

uncompressed size: 100MB

compressed size: 60MB

This yields:

Reading 100MB at 20MB/s from disk takes 5s.

Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.

In case the data is very small it's probable that seeking time will start dominating the times:

Reading 1MB at 20MB/s from disk takes 0,05s.

Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.

Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

I used to be a fan of all loose files ( especially for Oblivions clunky game engine ), but since reading that, and throughout the development of Vanilla Reduced Textures ( one of those BSAs has over 9000 textures compressed inside it replacing vanilla game textures ), I have found Ethatrons advice to be more optimal for my old laptop than loose files.

I have also seen mention of people testing with SSD drives who corroborate Ethatrons findings.

 

I do not have an SSD drive yet, so in my case it makes even more sense having a SATAII HD.

 

There is also the advantage that all files within a BSA do not become fragmented over time, so are quicker to access because there is less searching/thrashing of your HD to do, putting all the fragments of files together again for use. Regular defragging ofc will solve that, but if they are more optimally loaded from a highly compressed BSA .. its extra caretaking thats not required really.

 

 

Disadvantages to BSAs : As Kelmych mentions there are problems with repacking vanilla BSAs, but I am pretty sure that does not apply to Skyrim - Textures.bsa - Unless you repack the textures BSA and it exceeds the 2gb BSA file size limit that is ( and this being the S.T.E.P forum .. is a highly likely circumstance :) ).

 

I would not repack any other vanilla game BSAs though, string files being repacked are a problem, and also sounds being repacked cause problems too ( I think it varies depending on the compression ratio, probably a critical game load timing thing where the sound has to be decompressed from the BSA, and then decompressed from its fuz format ( which is also very highly compressed ) .. a fraction of time too long and it all goes wrong )

 

The official archiver I think I have seen mentioned has 7 compression levels, and by default uses its third ( something like that .. I dont use it )

Whereas BSAOpt / DDSOpt / NIFOpt has 10. The default being 10 maximum compression.

 

I do prefer loose files where it gets too complicated for me to visualize exactly what textures end up in game though, and like z, I like to use Wrye Bash's excellent file conflict resolution in those cases so I am sure of what ends up in game out of all the loose texture mods, which are sometimes not replacer textures ( custom textures which do not replace vanilla textures ) and can replace each other from different mods sets of textures ( Bodies male/female/beasts/Armour/Clothing in particular I like to keep a visual track of conflicts as loose files, to ensure which ones end up in game, and more importantly for me - at what resolution ).

[Kelmych]

Once Wrye Bash supports BSAs as an achive method, some of the advantage of archived loose files vs. BSAs will go away. The first two reported problems with BSAopt compression are for the two vanilla BSAs that contain audio data which, as alt3rn1ty mentions, can be problematic. We don't yet know exactly which resource is causing the recently reported problem with Crossbows in Dawnguard.

[PCG4m3r]

Similar to the issue Dark reported with Dawnguard, I ran into an issue related to Hearthfire today.

 

I was experimenting with the packing/optimizing/repacking using BSAopt/DDSopt after reading the Guides today. I actually did this with Dawnguard, Dragonborn, and Hearthfire, but immediately I noticed an issue with Hearthfire. So I restored the original DLC BSAs back to \Skyrim\Data and the issue went away.

 

Here is what I did and the result:

 

 

1. Unpacked all files from HearthFires.bsa to a temp folder using BSAopt 1.6.3

2. Used DDSopt 8 pre Update 4 with the custom DDSopt.ini to optimize DDS files and also constrained normals to 1024x1024

3. Copied optimized textures to the temp folder in step 1 and overwrote all applicable textures

4. Used BSAopt on all files in the temp folder to repack into new, optimized HearthFires.bsa file

5. Backed up my original HearthFires.bsa and replaced with the new optimized file

6. Went into the game without any mods loaded and only loaded the DLCs, but no HRDLC1/2/3

7. Had a level 1 character so went to console and typed "player.advlevel" to Leve up to 14 (just to get to 10 or higher to start Dawnguard quest)

8. Immediately a Courier came running up to me about seeing the Jarl of Falkreath and handed me a letter

9. I opend the letter and all it had at the top was "LOOKUP FAILED!"

10. I saved the game at that point with the letter in inventory

11. I exited and copied the original HearthFires.bsa back over

12. Launched game again, opened letter, and this time I could read it -- no error! 

 

 

I confirmed it was related to Hearthfires.bsa by copying the optimized back and overwriting the original. As soon as I loaded the game, the letter showed "LOOKUP FAILED!" again and problem cleared when I launched with the original .bsa again. I guess for now I'll stick with using the original/unoptimized DLC .bsa files and will just load optimized textures as loose.

 

I also followed the BSA Guide to convert these 6 BSAs using BSAopt, but I'm worried that there could be problems later on. Is it still recommended to convert these?

https://wiki.step-project.com/Guide:BSA_Extraction_and_Optimization#tab=Optimizing_Vanilla_BSAs

 

Skyrim - Animations.bsa

Skyrim - Interface.bsa

Skyrim - Meshes.bsa

Skyrim - Misc.bsa

Skyrim - Shaders.bsa

Skyrim - VoicesExtra.bsa

[Kelmych]

There are no reported problems with the 6 non-texture BSAs you listed. The optimized versions of these have been in use for quite a while.

 

I converted all the addon BSAs including the Hearthfire BSA to loose files and, after DDSopt optimization of the textures, I converted them back to BSAs using BSAopt. I received the letter from the Jarl and there isn't any problem with it in my game. I don't have one of the Dawnguard Crossbows so I have not been able to confirm the problem that alt3rn1ty mentioned.

 

I'll change the guide to mention the problems so users can choose if the want to use an archive of the loose files for the addon DLC BSAs if the textures are optimized. I'll also add a note so that users understand that if they use an archive of loose files, then all the BSAs in all the mods being used also need to be extracted so the mod uses an archive of loose files. This includes all of the unofficial patches as well any other mod being used, at least those might overwrite one of the vanilla resource files that came from an addon DLC that is not being used as a BSA. I do this anyway myself, but I want to make sure other users are aware of this.

[PCG4m3r]

Thanks Kelmych.

 

That's really odd, I wonder why the repacked BSA is causing a problem for for me? :?

 

Are you using BSAopt v1.6.3 or the 2.0 beta? 

[Dark]

Did you try taking the loose files from Dawnguard.bsa after DDSopt optimized the textures and creating an archive file with 7zip or another tool? I'd be interested to see whether this eliminates the problem.

 

There have been other problems noticed with using BSAopt to repair some (but not all) of the non-texture vanilla BSAs and then return them to a BSA format. The problem didn't occur at all with Skyrim 1.6 but does with Skyrim 1.8; the exact same BSAs didn't have any problems being decompressed in Skyrim 1.6 but had problems in Skyrim 1.8. It might be a problem with the BSA decompression code in Skyrim 1.8 when it is processing BSAs compressed using the improved compression algorithm that BSAopt and DDSopt use (even though Skyrim 1.6 had no problems with this).

 

No crash when using the loose files from an optimized Dawnguard.bsa.

 

I also tried using level 7 compression for BSA - still crash.

Looks like that Skyrim just didn't liked the BSA optimization (i.e. file duplication removal) that BSA/DDSOpt do, but it can't be tested, as there is no option to turn this feature off.

[lelec]

A quick question:

'update.bsa' is never directly mentioned in either 'BSA Extraction Guide' or in the 'DDSopt Guide', what to do with it ?

[SSL]

Extract textures form it, put them right below vanilla and above HDDLC

[alt3rn1ty]

Just a thought ..

 

For anyone experiencing crashes after creating just texture BSAs' -

 

If using BSAOpt or DDSOpt : You are changing the Game menu to Skyrim before creating the BSAs ?

 

If not then that alone will cause crashes, the BSA will be in the wrong format.

[lelec]

Thanks SSL.

[z929669]

@alternity: I guess you misunderstood my point. I don't use Skyrim - Textures.bsa anymore since I've extracted it and installed the optimized textures as loose files. If a mod that has it's textures in a bsa gets loaded' date=' will it still overwrite the loose files of Skyrim - Textures.bsa with it's textures or not? I'm not sure whether the new system that loose files win over bsa only applies to official Bethesda bsas or also to all other bsas...[/quote']

The BSAs and You article says the loose resources always win over those in any BSAs: "If the file exists in the Data folder' date=' it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA.". [/quote']

Then why does it make any sense to not repack the DDSopted textures pack into bsas I ask myself... I don't think I'll be unpacking all the bsas in my install just because of a "because that's what you can do to gain no measurable profit in performance but theoretically you do!"

I ddsopted my skyrim install all day long.

And what did my benchmark show?

 

 

 

 

Nothing. No improvement AT ALL.

Jesus, I've wasted enough of my life today, I'm going to bed.

The performance game is almost strictly in terms of load times, so FPS and VRAM will not be affected. However, you will notice an increase in speed mainly during Skyrim init and moving between cells in game.

 

This gain is especially helpful for HDD (but I still use this on my SSD, since I can still avoid processor usage burden and time by avoiding decompression)

 

OK, based on alt3rn1ty's conveyance of Ethatron's testing, I can see now that BSAs will often enhance load times, especially with HDD (my experiences must have always been placebo ... erroneous placebo, too :P )

 

However, I prefer the enhanced conflict resolution of loose files --and the ability to control outcome. Using an SSD makes loose files even more sensible, since defrag is not an issue (my brain cannot perceive millisecond diffs between data accesss from SSD versus from decompression from SSD anyway).

[PCG4m3r]

Just a thought ..

 

For anyone experiencing crashes after creating just texture BSAs' -

 

If using BSAOpt or DDSOpt : You are changing the Game menu to Skyrim before creating the BSAs ?

 

If not then that alone will cause crashes, the BSA will be in the wrong format.

Hey Alt3rn1ty,

 

I made sure Skyrim was the selected game when I had used BSAopt to unpack/repack. Of course, in my case I didn't experience any crash, but just a strange issue with some Hearthfire courier letter not working right. I don't know what else it could be, but it worked for Kelmych. Any ideas?

[Kelmych]

The performance game is almost strictly in terms of load times, so FPS and VRAM will not be affected. However, you will notice an increase in speed mainly during Skyrim init and moving between cells in game.

 

This gain is especially helpful for HDD (but I still use this on my SSD, since I can still avoid processor usage burden and time by avoiding decompression)

 

OK, based on alt3rn1ty's conveyance of Ethatron's testing, I can see now that BSAs will often enhance load times, especially with HDD (my experiences must have always been placebo ... erroneous placebo, too :P )

 

However, I prefer the enhanced conflict resolution of loose files --and the ability to control outcome. Using an SSD makes loose files even more sensible, since defrag is not an issue (my brain cannot perceive millisecond diffs between data accesss from SSD versus from decompression from SSD anyway).

It also seems to me to be a tradeoff. Yes, BSAs provide faster loading but the decompression uses CPU cycles. Which is more important in Skyrim, reducing CPU load associated with decompression or decreasing the load time (and perhaps some CPU loading, and chipset loading) for the resources?

[z929669]

There is a conversation on another thread around here (found it: Hardware thread beginning here) that is attempting to debunk the Skyrim-CPU dependency idea ... but it would seem that reducing the CPU clocks for any task would be overall beneficial ... ?

[Kelmych]

There is a conversation on another thread around here (found it: Hardware thread beginning here) that is attempting to debunk the Skyrim-CPU dependency idea ... but it would seem that reducing the CPU clocks for any task would be overall beneficial ... ?

I have two monitors, and on the 2nd monitor are (among other things) a CPU/RAM monitor and a GPU/VRAM monitor so I can always see what is happening with these. So far I've never seen CPU usage get above roughly 70% in Skyrim. I've had cases where it seems to be running out of RAM (I get an error warning); when that happened memory use was 3.2 Gb or less vs the 4Gb that is available to Skyrim.