User:Kelmych/Benchmark Testing: Difference between revisions

Revision as of 02:57, January 28, 2013

The following table provides means ± standard deviations for 4 parameters being used for system benchmarking with STEP mods.:

GPU temperature
GPU load
Dedicated VRAM (on-card memory) - This caps at 2 Gb on my system.
System VRAM (swapped-to-system memory) - lower is better, I am still determing the value at which moderate/severe stuttering on my system
FPS

In addition, the table includes two additional parameters:

maximum dedicated VRAM use, and
maximum dynamic VRAM use.

The last two reported values are the maximum of a five second running average of dedicated and dynamic VRAM, and are provided since they assist in building conclusions about the test.

Prior to running the benchmark tests all of the Vanilla textures (textures in Skyrim - Textures.bsa) were optimized using DDSopt as discussed in the DDSopt Guide. Selected STEP mods were also optimized prior to the test. The selection used the STEP Mod Optimization table from the DDSopt guide. Any mod with at least one Yes in the first 4 columns was optimized using DDSopt. Note that this criteria excluded all the large environment mods in the Conflicting Graphics section of the STEP guide; the authors of these mods have already done substantial optimization of their textures. The mods used were those in the Extreme STEP list with one exception. Instead of using only Skyrim HD and Serious HD in the Conflicting Graphics section of the list, a mix of textures selected from 5 mods were used to provide the textures that these two mods would have provided. The additional mods used are Skyrim Realistic Overhaul, Langleys Wood Metal and Stone Workshop, and Riften HQ. The specific textures included were the ones Cestral chose from these 5 mods for Texture Pack Combiner.

Previously, using Extreme STEP mods (constrained to 2Kx2K when needed), I found that I had stuttering and even CTD problems in areas of Skyrim known to cause high VRAM loading such as the road from the Brewery to Whiterun and the steps leading away from Dragonseach. I wanted to explore options for reducing VRAM use so I could use Skyrim without the problems I previously encountered.

Being fairly impatient, I was initially interested in the results from benchmark tests before any mods were loaded and then after all mods were loaded. I could then determine roughly how much work remained in reducing VRAM load while running the actual Skyrim game. I also wanted to investigate two options to get some reduction in VRAM usage:

compressing normal maps on a selected set of textures since z929669 found this noticeably decreased his VRAM use - for this experiment I chose the normal maps from the environmental mod mix mentioned above along with those in Skyrim Floral Overhau;, and
the effect, which I expected to be fairly small, in using cleaned up versions of the Skyrim bsa resource files other than just the Skyrim - Textures.bsa which had already been optimized. The description on the BSAopt nexus page shows that processing these other bsa files with BSAopt provides noticeable file size reduction through its removal of duplicates, and better read times since BSAopt uses better compression that takes less time in Skyrim to expand than the compression used by Bethesda. I did not use BSAopt on Skyrim - Sounds.bsa since there is a known problem with changing the compression on this file.

With the initial results shown below, it looks like reducing the size of normal maps on selected mods has significant potential to get VRAM use down on my system, and using the BSAopt-cleaned versions of the normal Skyrim resources had a small effect - I was actually surprised it had as much effect as it did since I expected the main effect of using these would be on CPU and RAM use and not on the graphic processing.

There is still experimentation needed to determine which mods should have reduced size normal maps, and I want to add results for a few intermediate points in the STEP mod process. I also want to run some tests with more dynamics so I can get a better understanding of VRAM use in more stressing cases. I also felt that for my experiments the Maximum VRAM values were a little more useful than the mean/standard deviation pair. Comparing tests 2 and 3, for example, the mean dedicated VRAM was reduced in test 3 but the maximum was not.

Template:Notice Small

Assessment	GPU Temp. [?C]	GPU Load [%]	Dedicated VRAM [MB]	System VRAM [MB]	FPS	Max GP VRAM [MB]	Max Sys VRAM [MB]
Windows background	43 ± 0.2	0 ± 0.2	88 ± 0	89 ± 0	n/a	88	89
(1) Bench 1.F.2 INI Tweaks	58.5 ± 3.7	76.3 ± 17	692.2 ± 63.2	114.3 ± 0.5	59.9 ± 0.7	782	115
(2) Bench 2.M Gameplay	61.1 ± 2.0	89.9 ± 11.9	1647.2 ± 169.5	115.8 ± 1.4	55.9 ± 4.4	1908	117
(3) Bench 2.M Gameplay	58.9 ± 3.6	86.6 ± 18	1618.0 ± 179.1	116.5 ± 0.5	55.9 ± 4.6	1924.2	117
(4) Bench 2.M Gameplay	58.7 ± 3.9	87.9 ± 13.9	1,445.6 ± 177.8	118.1 ± 0.8	55.7 ± 4.7	1736	119
(4) Bench 2.M Gameplay	58.7 ± 3.9	88.3 ± 13.0	1,412.4 ± 182.1	112.7 ± 0.5	55.7 ± 4.8	1720.4	113

Note that the windows background numbers are included in the first rows. This should be subtracted from all of the data following as a proxy for resources consumed only by Skyrim.

Bench 1.F.2 INI Tweaks - Performed after 1.F.2, 2.2.1 Guide, Optimized vanilla Skyrim - Textures (not HD), no modifications
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, Optimized selected mods, 2k-constrained
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, BSAopt optimized Skyrim vanilla resources, Optimized selected mods, 2k-constrained,
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, Optimized selected mods, 2k-constrained, normal maps reduced to 50% for environment mod mix plus Skyrim Flora Overhaul
Bench 2.M Gameplay - Performed after 2.M Gameplay , Optimized vanilla textures including HRDLC, BSAopt optimized Skyrim vanilla resources, Optimized selected mods, 2k-constrained, normal maps reduced to 50% for environment mod mix plus Skyrim Flora Overhaul

@@ Line 19: / Line 19: @@
 Prior to running the benchmark tests all of the Vanilla textures (textures in Skyrim - Textures.bsa) were optimized using DDSopt as discussed in the [http://wiki.step-project.com/Guide:DDSopt_%26_Texture_Overhauls DDSopt Guide].  Selected STEP mods were also optimized prior to the test. The selection used the [http://wiki.step-project.com/Guide:DDSopt_%26_Texture_Overhauls/Mod_Optimization STEP Mod Optimization table] from the DDSopt guide. Any mod with at least one Yes in the first 4 columns was optimized using DDSopt. Note that this criteria excluded all the large environment mods in the Conflicting Graphics section of the STEP guide; the authors of these mods have already done substantial optimization of their textures. The mods used were those in the Extreme STEP list with one exception. Instead of using only Skyrim HD and Serious HD in the Conflicting Graphics section of the list, a mix of textures selected from 5 mods were used to provide the textures that these two mods would have provided. The additional mods used are Skyrim Realistic Overhaul, Langleys Wood Metal and Stone Workshop, and Riften HQ. The specific textures included were the ones Cestral chose from these 5 mods for Texture Pack Combiner.
-Being fairly impatient, for the benchmark tests I was initially interested in the results from benchmark tests before mods were loaded and after all mods were loaded. I could then determine roughly how much work remained in reducing VRAM load while running the actual Skyrim game. I also wanted to investigate two options to get some reduction in VRAM usage:
+Previously, using Extreme STEP mods (constrained to 2Kx2K when needed), I found that I had stuttering and even CTD problems in areas of Skyrim known to cause high VRAM loading such as the road from the Brewery to Whiterun and the steps leading away from Dragonseach. I wanted to explore options for reducing VRAM use so I could use Skyrim without the problems I previously encountered.
-* compressing normal maps on a selected set of textures that z suggested would likely have a major effect on VRAM use - for this experiment I chose the normal maps from the environmental mod mix mentioned above along with those in Skyrim Floral Overhaul, and
-* the effect, which I expected to be fairly small, in using cleaned up versions of the Skyrim bsa files other than just the Skyrim - Textures.bsa which had already been optimized. The description on the [http://skyrim.nexusmods.com/mods/247 BSAopt nexus page] shows that processing these other bsa files with BSAopt provides noticeable file size reduction through its removal of duplicates, and better read times since BSAopt uses better compression that takes less time in Skyrim to expand than the compression used by Bethesda. I did not use BSAopt on Skyrim - Sounds.bsa since there is a known problem with changing the compression on this file.
+Being fairly impatient, I was initially interested in the results from benchmark tests before any mods were loaded and then after all mods were loaded. I could then determine roughly how much work remained in reducing VRAM load while running the actual Skyrim game. I also wanted to investigate two options to get some reduction in VRAM usage:
+* compressing normal maps on a selected set of textures since z929669 found this noticeably decreased his VRAM use - for this experiment I chose the normal maps from the environmental mod mix mentioned above along with those in Skyrim Floral Overhau;, and
+* the effect, which I expected to be fairly small, in using cleaned up versions of the Skyrim bsa resource files other than just the Skyrim - Textures.bsa which had already been optimized. The description on the [http://skyrim.nexusmods.com/mods/247 BSAopt nexus page] shows that processing these other bsa files with BSAopt provides noticeable file size reduction through its removal of duplicates, and better read times since BSAopt uses better compression that takes less time in Skyrim to expand than the compression used by Bethesda. I did not use BSAopt on Skyrim - Sounds.bsa since there is a known problem with changing the compression on this file.
-(z's conclusions - mine are not here yet)
+With the initial results shown below, it looks like reducing the size of normal maps on selected mods has significant potential to get VRAM use down on my system, and using the BSAopt-cleaned versions of the normal Skyrim resources had a small effect - I was actually surprised it had as much effect as it did since I expected the main effect of using these would be on CPU and RAM use and not on the graphic processing.
-Notice below where the standard deviation (SD) is rather large for Dedicated VRAM when there the VRAM usage is at peak efficiency (operating at near maximum capacity). However, the SD quickly goes down as available VRAM shrinks. This is because VRAM from second to second has nowhere to go but down, when it previously had room to get much higher OR much lower. As Dedicated VRAM SD decreases, this should correspond to an increase in swapped (System) VRAM, which should beget stuttering or lag in gameplay.
+There is still experimentation needed to determine which mods should have reduced size normal maps, and I want to add results for a few intermediate points in the STEP mod process. I also want to run some tests with more dynamics so I can get a better understanding of VRAM use in more stressing cases. I also felt that for my experiments the Maximum VRAM values were a little more useful than the mean/standard deviation pair. Comparing tests 2 and 3, for example, the mean dedicated VRAM was reduced in test 3 but the maximum was not.
-As the GPU becomes VRAM taxed:
-# FPS goes down
-# Mean-dedicated VRAM consumption increases
-# Mean-dedicated VRAM SD increases at first and then decreases as if fills to capacity
-# Mean-system VRAM increases
-# Mean-system VRAM SD increases rapidly as it is relied upon in varying degrees and then decreases as it fills to capacity (or the swap capacity of the GPU)
-At the point that the System VRAM SD begins to noticeably decrease, stuttering should become significant, as this indicates a very high swap frequency.]
 {{Notice Small|The numbers in the first 4 columns of the table represent [mean] ± [standard deviation]. If a Normal  distribution statistical model is a good fit to this data then 67% of all measures would be within the range of M ± SD, and 95% of all measures would be within M ± 2SD, and 99% would be within M ± 3SD. Because of the nature of the underlying physical processes, Normal distribution models are often reasonable for predicting values near the mean but poor models of extreme (very unlikely) values. Some care must be taken when using means and standard deviations to make conclusions across multiple tests involving situations that only happened infrequently during the test.}}
 {| class="sitetable"