STEP:Mod Testing

Template:Warning Template:Notice Mod Testing for S.T.E.P. -- by the S.T.E.P. Team

Updated: 3:08:29 29 July 2013 (UTC)

GUIDE FORUM THREAD

Introduction

Mod Testing Prep

Mod Testing Setup & Profiles

External Testing Procedure

In Game Testing Procedure

Save Game Archive

In-Game Utilities

External Tools

Notes/ Ideas

[edit]

General Information

This guide will attempt to lay out best practices for testing mods marked for testing which could be potentially included in S.T.E.P. With a guide as complex as S.T.E.P., it is beneficial to reduce as much variation as possible in mod testing, since variation can breed a host of issues. This guide will outline standardized testing techniques for S.T.E.P. mod testing. The approach will be bottom-up, meaning recommendations will begin at the hardware level, progressing to software, and then to mods.

Overview of Testing Procedure

This will be updated as the procedures are developed and outlined within the other tabs. Testing is separated into two sections: external testing and in-game testing.

• Read the thread OP and Nexus page descriptions of the mod and make pertinent notes. Note an potential conflicts with other STEP mods or issues with the mod itself.
• Read the Bethesda, Nexus, and Steam forums to develop a sense of user satisfaction and a list of possible bugs to attempt to confirm or deny.
• Examine the documentation of the mod, note any lack of clarity, installation and uninstallation directions, and general completeness.
• Validate the archive package, naming scheme, directory structure, and installation procedure.
• Use TES5edit to determine quality of edits in any plugin files and the scope of mod. Note any issue of not carrying over changes from the DLCs and/or from the Unofficial Patches.
• DDSopt used to determine quality of texture compression for texture mods.
• Examine files in game (to accomplish this we need to research a quick and easy way of identifying the actual name of the (texture, mesh, etc) file corresponding to what we see in game.)
• Examine script source of mods for reliability and compatibility (if not available ask the author).
• All mods with scripts must include methods for installing and uninstalling in-game or at the very least stopping scripts with a console command.
• Review Papyrus logs for script issues (if applicable).
• Each mod is examined in multiple ways in game to confirm functionality and compatibility.
  • Test in various key spots of the world using savegames.
  • Test in various key points of quest development using savegames.
  • Test with a Vanilla Skyrim installation. (if there are issues with the mod under this test then no further testing is needed)
  • Test with a Baseline STEP installation that is complete and current (All Baseline mods installed. All mods up-to-date. No non-STEP mods installed.)
  • Test with a new game and with adding it to an existing game with Baseline STEP.
  • Note quality changes to the game's Appearance, Gameplay, etc.
  • Note quantitative changes like VRAM, GPU, CPU, RAM, and FPS usage.
• Complete the Mod Testing wiki form and submit.
• Contact mod authors with hard evidence and constructive criticism to improve any area's in need of improvement before becoming a STEP recommended mod.

[edit]

This page has been streamlined. More or reviewed information will be added/edited.

General Information

A stable computer and Skyrim installation is crucial to testing a new mod. Without a stable computer and installation, it will much harder to determine if the mod itself is incompatible/buggy with other mods/Skyrim or if the mod is incompatible with your computer. Below you'll find some guidance on how to increase computer and installation stability, as well as how to setup your computer for mod testing.

Computer Stability

Reference the Skyrim Installation Guide as needed ... this is required reading for all mod testers. Peruse (or post questions to) the mod testing forum ... guessing is not standard procedure.

Device Driver Updates

A driver is a computer program that allows other computer programs to interact with hardware (device). Most drivers are updated relatively frequently, especially drivers for graphic devices such as AMD's or nVidia's graphic cards. Driver updates fix bugs and offer performance improvements. As such, it is quite important to have an up to date set of drivers for all your computer's hardware devices.

If your computer is pre-built, for example, you bought an off the shelf computer under a brand (vendor) name such as HP or ACER, a program(s) is usually included that assists in vendor specific driver updates. If this is the case, it is important to use this program(s), as many brand name off-the-shelf computers have drivers specifically designed for their hardware devices, rather than generic drivers. Consult the manual or the website for information on how to do this on your computer.

If your computer is custom made, then there are several excellent programs to assist with this. A great tool is Driver Genius, which will download an up-to-date list of drivers for almost all hardware devices, scan your computer with that list, and specify which need to be updated. The freeware version will do all of this and you can manually search for the drivers using this summary. If you upgrade to the paid version, then the program will also download and install the drivers for you. However, it is not recommended to allow this program to update your graphics drivers. This is best done manually by downloading them from AMD or nVidia.

Overclocking

If you are currently overclocking your computer it is recommended setting everything to its default/stock settings. This goes for the CPU, GPU, and RAM. Although the chances are low, it's possible for a new mod that is resource intensive to instigate stability issues, even if your previous overclock settings were stable. CPU and RAM overclocks can be managed from your motherboard's BIOS, and GPU overclocks are managed by a variety of programs. A recommended program for GPU overclock management is MSI Afterburner.

Ghosts in the Machine

Over time a computer will generate many redundant files that at the minimum will do nothing harmful, and at worst will cause stability issues. These files can include temporary files and redundant registry keys. It is recommended to use a program such as CCleaner to clean up these files, as well as to fix registry issues.

To ensure proper registry key cleaning, it is recommended to restart your computer after any program installation or uninstallation and any computer driver related changes. This is important because Windows will do some registry cleaning of its own when the computer restarts. Once you have restarted your computer, you can safely clean the registry as well as remove redundant files.

Disk Fragmentation

On hard disk drives (HDD), over time files that are regularly used by the computer will fragment (located on separate parts of the disk). This increases read times and potentially causes stability issues. Defragmenting alines fragmented files by placing them in order on the disks. It is always good practice to make sure the computers HDDs are defragmented before testing a mod. Thus, defrag your computer after installing a new mod, especially large mods such as texture overhauls. A recommended program is Auslogics Disk Defrag, which allows full drive defragmenting as well as specific file and folder defragmenting.

If you use a solid state drive (SSD), please disregard the defragmentation advice detailed above. Defragmenting a SSD offers no benefit and will significantly decrease the life span of the drive.

Background Processes

Background processes will take up resources, also potentially introducing stability issues when playing or testing Skyrim. A great program for eliminating as many of these background processes as possible is Game Booster. This will close unnecessary programs and background programs and services. Game Booster is optimized towards gaming so it does this very well and will restore your programs and services once you're done gaming.

TESV.exe Properties

These settings will ensure that when Skyrim runs, Windows Aero Theme will be disabled, freeing up VRAM, since Aero may use between 64 to 128MB of VRAM. To access these settings navigate to your skyrim folder, right click on TESV.exe, and then click on the Compatibility tab. Tick the Disable Visual Themes and Disable Desktop Composition boxes, as shown in the image below. This only applies to Windows Vista or later.

thumb:Figure 1. TESV.exe Properties Settings
 
 

Skyrim Stability and Settings

Reference the Skyrim Installation Guide as needed ... this is required reading for all mod testers. Peruse (or post questions to) the mod testing forum ... guessing is not standard procedure.

Launcher Settings

Choose Medium or High settings in the Launcher. Then select a screen resolution. If you can, try to select 1920x1080 or as close to it as possible, as this may be the most common, at 31%, screen resolution at the moment. However, if you are unable to do so, do not worry. Full HD (1920x1080) users are by no means a large majority. Other common screen resolutions to test at are 1680x1050, 1600x900 (laptop users), 1366x768, and 1280x1024. AA should be set to at least 4x, and AF to at least 8x. All view distances can be maxed out. Distant object detail should be high or medium, FXAA disabled (unchecked), and all water reflection options should be disabled (unchecked).

INI Settings

It is recommended to test a mod on medium or high settings, and once verified, test on ultra if possible. Testing on ultra may skew the results of the testing, especially if others may want to use the mod. The majority of Skyrim users will not have high end computer setups which are able to run on ultra settings.

The easiest way to achieve this is to use Wrye Bash's INI Edits function. Make sure you generate a default Skyrim.ini and SkyrimPrefs.ini. This can be done by deleting your current ini's (remember to back them up first) in your Documents folder. Once deleted, generate new inis by changing your settings to either medium or high in the Skyrim Launcher, as mentioned above. Once these default inis have been generated, back them up. Navigate to the INI Tweaks folder in Skyrim's Data folder. This folder will only be present if you have installed Wrye Bash. Create two new text files and name one Default Medium (or High) Mod Test Skyrim.ini, and the other Default Medium (or High) Mod Test SkyrimPrefs.ini. Make sure that your two text files change from .txt to .ini. Once these have been created, copy the contents of the default ini's into their respective ini files you just created. The same should be done for your normal/custom settings that you play Skyrim with, but name the resulting ini's differently, such as My Skyrim.ini, or My SkyrimPref.ini

This will allow you to easily switch between mod testing ini's and your normal ini settings. To apply the ini settings, right click on the ini file in Wrye Bash and chose apply. Be warned that if you would like to remove the applied ini settings, you cannot do so from Wrye Bash. For example, if you apply your normal ini settings to your mod testing ini setting, you will overwrite many ini settings. However, if your mod testing ini settings contain entries that do not exist in your normal inis, then these entries will remain. Only entries that exist in both will be overwritten. It is easier just to copy your settings from your normal ini files, delete the text inside your Skyrim.ini and SkyrimPrefs.ini, and paste the copied text inside if you wish to revert back to your personal settings. This can also be done from your backups.

uGrids

This ini setting MUST be set to the default value of 5. For those of you who use a value of 7 and above, be warned that these higher values are 100% inherently unstable. It may be stable as a rock for you, but please don't test a mod with anything but the default value of 5. The chance that a new mod may cause stability issues with higher uGrids values is quite high, and again, most Skyrim players will be using the default value. To easily switch between uGrids values, create the relevant ini files in the INI Tweaks folder for WB as mentioned in the section above.

Should you attempt to test a mod on a save game that used to have a higher uGrids value, you will be unable to load it. There are three solutions. The first is to start a new game, which is highly preferred. The second is to use a Bat file to revert to the default value. These can be obtained from the DNA INI Generator site. Once there, navigate to the green panel called Downloads, and follow the instructions there. The third option is to use a selection of vanilla saves and STEP complete saves that can be downloaded under the save game section.

Default uGrids Settings (Skyrim.ini)

[General]
uGrids=5
uExterior Cell Buffer=32

Papyrus Logging

These ini tweaks will allow you to see if mod issues are caused by errors in the mods scripts. Not all mods will contain scripts though. Again the easiest way to apply these is create a sperate ini file in Wrye Bash's INI Tweaks folder. This tweak is applied to the Skyrim.ini file. The log files can be found where your save games are, in the Scripts folder.

Papyrus Logging (Skyrim.ini)

[Papyrus]
bEnableLogging=1
bEnableTrace=1
bLoadDebugInformation = 1

Mod Tracing

At the moment, in game mod tracing is very basic. Currently for textures, you can open the command console by pressing the ~ key, and click on the texture that you are interested in. In the console command screen you will see its name and reference ID (eight hexadecimal characters, ie "001C8F02"). The name and/or the reference ID may be useful in tracing the object or texture of interest.
 
 

[edit]

This page is still under development. More or reviewed information will be added/edited.

Mod Managing Programs

The mod should be first inspected by a mod organizer program such as Wrye Bash (WB) or Mod Organizer (MO). The Nexus Mod Manager (NMM) is no longer supported by STEP. Using these programs will let you know if the mod is suitably constructed and whether it conflicts with other mods. (mods with BSA archives will have to be extracted as loose files to view conflicts in WB) At this point, the choice is yours as long as you use one or the other. Both have different merits; however, for testing both work just fine. MO might be the wiser choice if you don't want to use SIS.

Wrye Bash

Mod Organizer

MO creates virtual folders when installing mods, which means it will keep the Skyrim directory clean. This can help in troubleshooting. It can also create profiles, similar to SIS.

Skyrim Installation Swapper / Mod Organizer

These are essential utilities for mod testers.

To quote the author; "Skyrim Installation Swapper (SIS) let's you create up to ten individual Skyrim installation "profiles" which can be swapped in and out easily, via GUI or hotkey. For those familiar with Mod Organizer (MO), MO can be used to do the same thing.

The use of one of these utility is important regarding testing, because a mod must be tested against a vanilla Skyrim setup (no mods at all) and a full Baseline STEP modded Skyrim setup. These programs makes it much easier to switch between the two without the hassle of multiple re-installs.

Skyrim Installation Swapper

Mod Organizer

Mod Testing Setup

Reference the Skyrim Installation Guide as needed ... this is required reading for all mod testers. Peruse (or post questions to) the mod testing forum ... guessing is not standard procedure.

Overview

• Skyrim installed in a non windows directory, if able. Preferably something like E:\Games\Steam\steamapps\common\skyrim
• All DLC's installed and cleaned with TES5edit. (Just what you own, you don't need to buy stuff just to test)
• All Unofficial Patches installed (Just for the DLCs you own)
• All INIs with Vanilla Med to High settings.

General Setup

1. Ensure a clean install of Steam and Skyrim (non-windows directory)
2. Use TES5Edit to clean any relevant DLC content present
• Optional: use DDSopt to fix the vanilla textures
3. Install and properly configure utilities and extenders
• SKSE
• BOSS
• Wrye Bash and/or Mod Organizer
• TES5Edit
• DDSopt (Latest stable version. Also replace the INI with the STEP INI)
• Skyrim Installation Swapper (Not needed if you've chosen MO as your mod manager.)
• FRAPS (For performance benchmarking)
• Your preferred capturing program. STEP recommends the paid version of FRAPS, but no need to spend money to test unless you want to.
4. Start a new game, play through until you leave Helgen, and save as soon as the dragon flies away after exiting the cave.

SIS Profile Setup

1. Within SIS, name current #1 profile to "Single Mod Test"
2. Copy/save profile to #10 slot, naming as "Vanilla" using SIS
3. Copy/save profile to #2 slot, naming as "STEP Mod Test", then select as "Active Profile" using SIS
4. Install Core STEP mods expected in next release (list provided by admin in mod testing forum)
5. Become familiar with the mod to be tested:
• Read the mod-site description
• Carfully review any available documentation (i.e., within mod package and relevant mod forums)
6. In SIS, activate the Single Mod Test profile and install the mod to be tested using WB or MO. Provide feedback by completing the ModTest Form on the wiki (in development)
7. In SIS, activate the STEP Mod Test profile and install the mod to be tested using WB or MO. Provide feedback by completing the ModTest Form on the wiki (in development)

Mod Organizer Profile Setup

To be added...

[edit]

Below is the outline of what will fill this tab as the content is developed.

General Information

Now that your system is setup for mod testing, the fun begins! External testing is where all testing should start.

External Mod Testing Procedure

Step 1 - Opening Post/Workflow

Read the mod's thread opening post (OP) or workflow in the Mod Testing Forum.

There you will find an outline of what needs to be tested for the particular mod. Make note of this and use it when testing begins.

Step 2 - Nexus Page

Read the Nexus Page in it's entirety

Read the mod's Nexus page description completely as well as the changelog (if provided). Make note of any special installation/uninstall instructions, of any known issues with the mod or mod compatibility issues, and of any potential conflicts with the DLCs and/or other STEP mods. The changelog can often give you a sense of where the author is taking the mod and what you might be able to expect from a mod and it's author.

Step 3 - Mod's Forums

Read the Bethesda, Nexus, and Steam forums

A complete read is not necessary; however, try to develop a sense of user satisfaction and a list of possible bugs to attempt to confirm or deny. This is also a good way to find out how active the author is with the mod. No word from the author in a long period of time usually indicates they have been absent and may no longer be actively supporting the mod.

Step 4 - Documentation

Examine the documentation included with the mod

Read the Readme and any other documentation which comes with the mod. Make note any lack of clarity, installation/uninstall directions, and general completeness. If the mod does not include documentation, please make note of this as well for your review.

Step 5 - Validation

Validate the archive package, naming scheme, and directory structure.

Step 6 - DDSopt

Used DDSopt to determine quality of texture compression for texture mods.

Step 7 - TES5Edit

• Inspect in TES5Edit to determine quality of edits in any plugin files and the scope of mod

• Make note of any issue of not carrying over changes from the DLCs and/or from the Unofficial Patches and note any conflicts with mods already in STEP. (Neo's process of doing this with SR would be helpful here)

Step 8 - Inspection

• Inspect in WB/MO

• Make note of any file conflicts

Step 9 - Installation/Uninstall

• Validate the installation procedure
• Validate the uninstallation procedure

• Note any leftover files which the uninstallation failed to remove

[edit]

Still to be reviewed.

General Information

This section is the most important and will detail the steps required to test a mod in such a way that it can be recommended for STEP. It will be as simple and streamlined as possible. However, regarding the complexity, breadth and depth of mods for Skyrim, this guide will never be able to cover all testing scenarios. Ultimately, this guide should cover the majority of mod testing scenarios, but when it doesn't use your best judgment.

Texture Mod Testing

Testing Procedure

To begin with, the mod you have chosen to test will be examined using several tools to assess viability.

Next, the mod will be tested on a vanilla Skyrim profile.

Thirdly, the mod will be tested on a complete Baseline STEP Skyrim profile.

Lastly, the Mod Testers will write detailed review (as much as possible) about their findings.

Screenshots/Video

When screenshots or videos are needed for comparisons, provide them from in-game sources. Do not use "studio" applications to provide shots and/or video. In-game lighting conditions and rendering can change the appearance of many textures compared to studio applications. Therefore, it is very important to capture these comparisons from within the game itself.

FRAPS is an excellent program for both screen and video captures in-game as well as capturing FPS data; however, feel free to use your preferred program for captures. The paid version of FRAPS provides more file formats for screen captures and PNG is the preferred format due to it's accuracy in capturing correct colors. JPEG can be used; however, some loss of correct coloring can be lost. Do not use GIF format! For video capture, AVI or MP4 file formats are best. FRAPS will capture videos in the AVI format.

When uploading captures for posting compares on the STEP forums, please use a 3rd party service. Do not store your captures on the STEP wiki! Imgur has proven to be an excellent free image hosting site for uploading screen captures to. For video captures, please use YouTube. Other sites have proven annoying for members to use, most requiring an account to view the videos. YouTube videos can also be included right in the post using the video clip button in the formatting bar on the forum.

External Assesment

Mod Package Structure Assessment

Using your mod manager or 7zip, observe if the downloaded mod package is properly structured or configured for installation. Detailed information on proper mod package structure can be found here.

Conflicts & Overwrites Assessment

Using your mod manager, take note of any conflicts that appear against vanilla Skyrim files. The process is different for each mod manager so you should be familiar how to do this in your chosen manager. For mods with BSA archives, you'll need to extract these archives into loose files before you'll be able to see the conflicts. This can be done easily using BSAopt.

Plugin Assessment

Using WB, assess the plugin. Right-click on the .esp or .esm plugin and highlight "Mod Cleaning" in WB. In the drop down box, select "Scan for UDR's". This will indicate if the mod is clean or dirty, which should be noted in the review. Once additional programs such as TESVEdit, TESVJedit, and TESVGecko are released a more detailed explanation will be added. This can also be done with BOSS; however, neither of this methods are completely accurate.

Enable a Vanilla Skyrim Profile

Enable your vanilla version of skyrim using SIS or MO. Do not start the game yet.

Generate Default Skyrim.ini & SkyrimPrefs.ini Files

If you have not already done so using SIS or MO, generate default ini files on medium or high settings with the relevant modifications by following the instructions in the Skyrim Stability and Settings.

Papyrus Logging

During all stages of mod testing, papyrus lodding should be enabled. Activate it by changing the relevant ini values in the Skyrim.ini file under Documents/My Games/Skyrim/ (for Win7).

Testing Locations

• Testing Hall

• Riften

• Whiterun

• Riverwood

• Mod Specific

Helpful Console Commands

To open the in-game console press the [ ~ ] (tilde) key normally located just below the ESC key. Press it again to close the console.

coc locationName

Transports you to specified location. Replace the locationName with the name of the location. A list of location names can be found here.

tgm

Toggles God Mode on/off.

tcl

Toggles collisions on/off. Don't use while falling or CTD can occur.

tfc

Toggles Free Camera Mode on/off

tfc 1

Pauses Free Camera Mode

tm

Toggles menus on/off. Useful when taking screenshots.

[edit]

Still to be created and archives added.

Vanilla Saves

S.T.E.P. Complete Saves

[edit]

Still to be reviewed.

General Information

Several in-game tools will be specified here. These tools will for the most part be mods that are particularly useful when it comes to mod testing.

Elys MemInfo is a SKSE plugin that is very useful for displaying in game resource usage. Currently it is able to show RAM, Pagefile, Virtual Manager, Handles, and VRAM use and utilization by Skyrim. This is most useful when testing mods that affect graphics, such as texture mods, FXAA Injectors, ENB presets, and lighting and shadow tweaks and mods. For example, when testing out texture omptimizer, such as DDSOpt, it is possible to see how much VRAM use has decreased in certain areas. If you are hitting your graphic cards VRAM limit, it is useful to know that the optimizations have lowered VRAM use. Since Elys MemInfo displays this information in game, there is no need to minimize the game to bring up alternative resource use tools such as GPUZ or the Task Manager. (Requires SKSE)

Alternative Start - Live Another Life is an extremely useful in game testing mod. As explained before, mod testing is best done by starting a new game. One of the issues with testing out one, or many, mods quickly when starting out a new game, is the requirement by Skyrim to watch and play the opening sequence of the game. This mod allows mod testers to simply skip the rather long introduction sequence. For reducing the time required for testing mods, Live Another Life is an invaluable addition. It is also a safe and clean mod, and will not break any quest lines.

No Boring Sleep Wait Menu allows the tester to wait up to 31 days quickly and easily, which is not possible using the vanilla waiting system. Skyrim requires a 31 day in-game wait time before as many cells as possible can be reset. Cell resets are useful when mod testing for several reasons. A cell may be badly generated, or you wish to install a mod without starting a new game for testing. In both cases a cell reset will potentially generate a proper cell, and the new mod may be integrated properly, especially if the mod affects spawns.

ScenicCarriages transforms the non-moving Skyrim carriages to a fully functional carriage ride. This mod is great for bench testing performance within the game. It will allow you to provide more accurate results than if you tried to bench and re-bench by simply walking/running. Simply pay the carriage driver to go from Whiterun to Riften, climb in the back of the carriage, hit your benchmark hot key (FRAPS) and you're off. For best results when using this for benchmarking, use the default settings and don't move the mouse to look around. The ride from Whiterun to Riften will take around 15 minutes and provides a wide range of textures during the benchmark.

[edit]

Still to be reviewed.

General Information

This page contains a list of programs that will be recommended for use during the testing procedure.

Generic Programs

GPU-Z is an amazing lightweight program that offers a plethora of ifnormation regarding any type of gpu you have, as well as allowing logging and graphing of gpu related processes such as vram load, temperature, clock speeds, and far more. This will be the main tool for testing graphic related mods, as its logging ability is very useful here.

Nvidia Inspector (NI), is a program that allows for in depth gpu information of Nvidia graphics cards. It allows for very fine profile tuning for skyrim, as well as monitors for a range of process such as gpu temperature and vram use.

FRAPS is an excellent program for capturing both screenshots and video. It's as easy as pressing your Hotkey while in-game. FRAPS is also great at capturing FPS data for performance testing. Both free and paid versions are available; however, if you're serious about Mod Testing and plan on capturing a lot of shots or videos then the paid version, which opens up new (and better) file formats for screenshots and better options overall, is well worth the cost.

FRAPS Bench Viewer is a free tool that graphs out your FRAPS logs from your benchmarks (FPS data) in easy to read visual graphs. This makes it very easy to spot performance issues.

ATI Tools

Skyrim Focused Programs

• Skyrim Performance Monitor
• TES5Edit
• TES5Dump

[edit]

To be added into the guide ...maybe.

• Camera mod

• DDSopt if testing texture mods, link to DDSopt guide.

• compatebility checking

• tes5edit, tes5gecko when released
• ck workarounds

• wrye bash

• esmfy esps to fix ctds/ tesvsnip.

• INI tweaks / WB ini tweaks

• medium default settings for stability

• maybe upload WB ini wteaks files.

• papyrus logging
• mod tracing (need to test)

• saves

• multiple save files from vanilla and STEP setup, heavy exterior areas, heavy interior.
• link to total cell clear in troubleshooting guide
• z92's suggestions for 60s run throughs.

• in game tools

• camera mod

• software tools

• skyrim performance monitor, FPS background booster.