User:Kelmych/Benchmark Testing

The following table provides means ± standard deviations for 4 parameters being used for system benchmarking with STEP mods.:

GPU temperature
GPU load
Dedicated VRAM (on-card memory) - This caps at 2 Gb on my system.
System VRAM (swapped-to-system memory) - lower is better, I am still determing the value at which moderate/severe stuttering on my system
FPS

In addition, the table includes two additional parameters:

maximum dedicated VRAM use, and
maximum dynamic VRAM use.

The last two reported values are the maximum of a five second running average of dedicated and dynamic VRAM, and are provided since they assist in building conclusions about the test.

Prior to running the benchmark tests all of the Vanilla textures (textures in Skyrim - Textures.bsa) were optimized using DDSopt as discussed in the DDSopt Guide. Selected STEP mods were also optimized prior to the test. The selection used the STEP Mod Optimization table from the DDSopt guide. For all mods used in the testing, any mod with at least one Yes in the first 4 columns was optimized using DDSopt. This includes the STEP mods, the mods from Skyrim Revisited, and other mods I use. Note that this criteria excluded all the large environment mods in the Conflicting Graphics section of the STEP guide; the authors of these mods have already done substantial optimization of their textures. The mods used were those in the Extreme STEP list with one exception. Instead of using only Skyrim HD and Serious HD in the Conflicting Graphics section of the list, a mix of textures selected from 5 mods were used to provide the textures that these two mods would have provided. The additional mods used are Skyrim Realistic Overhaul, Langleys Wood Metal and Stone Workshop, and Riften HQ. The specific textures included were the ones Cestral chose from these 5 mods for Texture Pack Combiner.

Previously, using Extreme STEP mods (constrained to 2Kx2K when needed), I found that I had stuttering and even CTD problems in areas of Skyrim known to cause high VRAM loading such as the road from the Brewery to Whiterun and the steps leading away from Dragonseach. I wanted to explore options for reducing VRAM use so I could use Skyrim without the problems I previously encountered.

Being fairly impatient, I was initially interested in the results from benchmark tests before any mods were loaded and then after all mods were loaded. I could then determine roughly how much work remained in reducing VRAM load while running the actual Skyrim game. I also wanted to investigate two options to get some reduction in VRAM usage:

compressing normal maps on a selected set of textures since z929669 found this noticeably decreased his VRAM use - for this experiment I chose the normal maps from the environmental mod mix mentioned above along with those in Skyrim Floral Overhau;, and
the effect, which I expected to be fairly small, in using cleaned up versions of the Skyrim bsa resource files other than just the Skyrim - Textures.bsa which had already been optimized. The description on the BSAopt nexus page shows that processing these other bsa files with BSAopt provides noticeable file size reduction through its removal of duplicates, and better read times since BSAopt uses better compression that takes less time in Skyrim to expand than the compression used by Bethesda. I did not use BSAopt on Skyrim - Sounds.bsa since there is a known problem with changing the compression on this file.

Since the VRAM needs varied significantly over time during the benchmark, the standard deviation provides more information about the benchmark than the graphic card since during the test the maximum VRAM available on the card was not exceeded.

With the initial results shown below, it looks like reducing the size of normal maps on selected mods has significant potential to get VRAM use down on my system, and using the BSAopt-cleaned versions of the normal Skyrim resources had a small effect - I was actually surprised it had as much effect as it did since I expected the main effect of using these would be on CPU and RAM use and not on the graphic processing.

There is still experimentation needed to determine which mods should have reduced size normal maps, and I want to add results for a few intermediate points in the STEP mod process. I also want to run some tests with more dynamics so I can get a better understanding of VRAM use in more stressing cases. I also felt that for my experiments the Maximum VRAM values were a little more useful than the mean/standard deviation pair. Comparing tests 2 and 3, for example, the mean dedicated VRAM was reduced in test 3 but the maximum was not.

NOTE:

The numbers in the first 4 columns of the table represent [mean] ± [standard deviation]. If a Normal distribution statistical model is a good fit to this data then 67% of all measures would be within the range of M ± SD, 95% of all measures would be within M ± 2SD, and 99% would be within M ± 3SD. Because of the nature of the underlying physical processes, Normal distribution models are often reasonable for predicting values near the mean but poor models of extreme (very unlikely) values. Some care must be taken when using means and standard deviations to make inferences across multiple tests involving situations that happened infrequently during the test.

Assessment	GPU Temp. [?C]	GPU Load [%]	Dedicated VRAM [MB]	System VRAM [MB]	FPS	Max GP VRAM [MB]	Max Sys VRAM [MB]
Windows background	43 ± 0.2	0 ± 0.2	88 ± 0	89 ± 0	n/a	88	89
(1) Bench 1.F.2 INI Tweaks	58.5 ± 3.7	76.3 ± 17	692.2 ± 63.2	114.3 ± 0.5	59.9 ± 0.7	782	115
(2) Bench 2.M Gameplay	61.1 ± 2.0	89.9 ± 11.9	1647.2 ± 169.5	115.8 ± 1.4	55.9 ± 4.4	1908	117
(3) Bench 2.M Gameplay	58.9 ± 3.6	86.6 ± 18	1618.0 ± 179.1	116.5 ± 0.5	55.9 ± 4.6	1924.2	117
(4) Bench 2.M Gameplay	58.7 ± 3.9	87.9 ± 13.9	1,445.6 ± 177.8	118.1 ± 0.8	55.5 ± 4.7	1736	119
(5) Bench 2.M Gameplay	58.7 ± 3.9	88.3 ± 13.0	1,412.4 ± 182.1	112.7 ± 0.5	55.7 ± 4.8	1720.4	113
(6) Bench 2.M plus	59.3 ± 3.8	89.4 ± 12.7	1,482.3 ± 196.9	110.5 ± 0.6	54.7 ± 5.2	1818	112

Note that the windows background numbers are included in the first row. This should be subtracted from all of the data following as a proxy for resources consumed only by Skyrim.

Bench 1.F.2 INI Tweaks - Performed after 1.F.2, 2.2.1 Guide, Optimized vanilla Skyrim - Textures (not HD), no modifications
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, Optimized selected mods, 2k-constrained
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, BSAopt optimized Skyrim vanilla resources, Optimized selected mods, 2k-constrained,
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, Optimized selected mods, 2k-constrained, normal maps reduced to 50% for environment mod mix plus Skyrim Flora Overhaul
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, BSAopt optimized Skyrim vanilla resources, optimized selected mods, 2k-constrained, normal maps reduced to 50% for environment mod mix plus Skyrim Flora Overhaul
Bench 2.M plus - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, BSAopt optimized Skyrim vanilla resources, Optimized selected mods, 2k-constrained, normal maps reduced to 50% for environment mod mix plus Skyrim Flora Overhaul, almost all the mods in Neovalen's Skyrim Revisited, plus additional mods that I use; selective optimization was used for all the additional mods that were not in the previous test