Mod Testing Guide
Mod testing for the Step Guides
by: The Step Team | Forum Topic
WIPThis Guide is currently outdated and requires updating to current processes/standards. It will be worked on over time.
It's beneficial to reduce as much variation as possible during mod testing, since variation can cause a host of issues and inconsistencies. Therefore, this Guide will lay out standardized techniques for testing mods. Step Modifications uses this process for all mods used within the official modding guides. Recommendations are given from at the hardware level up; progressing through software and setup, and then to mod testing.
What is Mod Testing?
Mod testing can be many different things; however, for the purposes of this Guide, mod testing is the testing of mods for inclusion to a mod list. This involves testing a mod's stability against other mods in the mod list, testing the mod's function to make sure it's working as intended, comparing it to other mods of similar content to find the best solution, and in the case for Step Modifications, checking it against Elder Scroll lore to make sure the mod is lore-friendly and meets the Step Mandates. Step Modifications has staff members, titled Curators, whom this job falls to, however, all staff usually do some amount of mod testing.
Mod testing can be long and tedious work at times, but it can be fun and enjoyable too. This is to say, mod testing is only for those that have a real dedication to it. As such, the turnover rate on users testing for projects tends to be rather high since these users can quickly become "worn out", if they don't realize what they've gotten themselves into.
To become a Curator, see here:
Overview of Testing Procedure
Each mod is examined in multiple ways in game to confirm functionality and compatibility. This is an overview of what the mod testing process involves:
- This is non-negotiable for all Step Curators.
- tested in relevant locations of the game world
- If a mod hasn't been tested in-game, it should be clearly stated so when posting reviews.
- tested in relevant quests
- tested against other mods from relevant mod lists
- tested with a new game and/or with adding the mod to an existing game (as required)
- noted changes to the game's Appearance, Gameplay, etc.
- noted quantitative changes like VRAM, GPU, CPU, RAM, and FPS usage
- review posted of the mod with personal findings and supporting data such as screenshots, benchmarks, etc. to the mod's forum topic
If issues are found, it's recommend to be courteous and contact mod authors with evidence and constructive criticism on how to improve any areas of the mod with issues or in need of improvement.
A stable system and game installation are critical before mod testing can begin. Without a stable computer and game installation, it will be much more difficult to determine if the mod itself is incompatible with the mod list, or if the mod is incompatible with the system it's running on. Testers should have their systems set up already from following one of the Step modding guides. However, below are some general reminders and best practices in preparing the environment for mod testing.
- Ensure the System Setup Guide is complete.
- Ensure a clean and pure installation of the modding guide for which the testing is being completed for is complete.
- Disable anything that isn't related to the guide; the installation should be 100% pure.
All game INI and launcher settings should be set according to the modding guide for which testing is being performed. Meaning any post-installation changes should be reverted back to the guide's defaults.
Mod Organizer Profile Setup
Step's only officially supported mod manager is Mod Organizer. Mod testing is where it proves itself, again, more helpful than any other manager. MO's feature set speaks for it, however, for mod testing it has some key advantages:
- Mod asset conflict viewer
- A virtual Data folder means the game directory stays completely clean.
- Ability to create multiple profiles without the need for a third party app.
- Nexus integration for downloading, installing, and updating.
- Mod version tracking and notification when mods are out-of-date.
- Profile specific INI files. No need to keep track of multiple INIs for multiple play-throughs.
Follow these instructions to set up the profiles in Mod Organizer for testing:
- Create two new profiles in MO for testing:
- Copy the Step Guide profile for each new profile and name them: Vanilla & GuideName Testing
- Make sure the Local Savegames box is not checked for the testing profiles.
- Make sure the Automatic Archive Invalidation box is selected for the testing profiles.
- Copy the Step Guide profile for each new profile and name them: Vanilla & GuideName Testing
- Select the Vanilla profile and ensure no mods are enabled in MO.
- Start a new game from this Vanilla profile
- Play through any introductory quests until the first point is reached that allows for free roaming of the game (e.g. for Skyrim play until leaving Helgen cave).
- Once at a free roaming stage, save the game.
- This save is the vanilla save from which to test new mods, so don't overwrite it!
- Tip: Use the console, if available, to give the save a custom name for quick reference.
External Testing Procedures
External testing is where all testing starts. Use the following steps as a guide to testing any mod externally before moving on to testing in-game. Each step should be completed and notes taken from each before in-game testing occurs.
Step 1 - Opening Post/Workflow
Read the mod's topic opening post (OP).
- Here, an outline of what needs to be tested for a particular mod can be found. Make note of this and use it when in-game testing begins. If nothing is outlined in the OP, take steps to establish an outline for testing by editing the OP. This outline should include what mod options should be tested, how to test, steps to take to make the mod compatible with other mods, etc.
Step 2 - Nexus Page
Read the Nexus Page in its entirety.
- Read the mod's Nexus page description completely, as well as, the changelog (if provided). Make note of any special installation/uninstall instructions, any known issues or mod compatibility issues with the mod, and of any potential conflicts with the DLCs and/or other mods within the Step Modification Guide. The changelog often provides a sense of what direction the author is taking the mod and what might be expected from a mod and its author in the future; as far as how often they fix issues, features being adding/edited, etc. Update the OP on the forums with any relevant information.
Step 3 - Mod's Comments
Read the Nexus comments.
- A complete read is not necessary; however, try to develop a sense of user satisfaction, and a list of possible bugs to attempt to confirm or deny (bugs can also be listed in the page's Bug tab, when provided). This is also a good way to find out how active the author is with the mod. No replies from the author over a long period of time usually indicates they have been absent for a while and/or may no longer be actively supporting the mod. Unless a mod is fully developed with few to no bugs being reported, some sort of interaction from the mod author is preferred to ensure the mod isn't abandoned.
Step 4 - Documentation
Examine the documentation included with the mod.
- Read the Readme and any other documentation that comes with the mod. Note any lack of clarity, installation/uninstall directions, and general completeness. If the mod does not include documentation, please make note of this as well for the final review.
Step 5 - Validation
Validate the archive package, naming scheme, and directory structure using MO or 7zip.
- Observe if the downloaded mod package is properly structured and configured for installation or not. Mod Organizer can also use BAIN type installations; however, FOMODs are preferred which use XML. FOMODs can be validated using an online validation tool, XML Validator, or users can use FOMOD Validator.
Step 6 - xEdit
Inspect the mod in xEdit to determine quality.
- Note any issues of the mod not carrying over changes from DLCs and/or from the Unofficial Patches. Also note any conflicts with mods already in the Step Modifications Guide. Knowledge of xEdit and conflict resolution will be required for this step. Add relevant information to the OP on the forums.
Step 7 - Installation/Uninstall
- Validate the installation procedure listed on the Nexus page.
- Validate the uninstallation procedure listed on the Nexus page.
Step 8 - Inspection
- Inspect in MO
- Using Mod Organizer, take note of any asset conflicts that appear against vanilla files and other mods in the Guide. Also extract any BSAs to ensure the conflicts are being read. BSAs should not remain extracted unless comparisons are needed for things such as textures. Otherwise, remove the extracted BSA contents before continuing with testing.
- Run LOOT
- Unless the mod is new on the scene, LOOT will recognize it and provide some valuable information about the mod such as if the mod is clean or dirty, requires other mods, etc. Include this information, if any, in reviews.
Step 10 - Mandate
After reviewing the mod and before in-game testing occurs, the final step is to pit the mod against the Step Modification Mandate. To do this, review what Core and Extended are and are not about. Ensure the mod does not fall into the "...not about" sections. If it does, no more testing is required. Review and post your findings why the mod doesn't fit the Mandate. If the mod passes this step, continue on to testing the mod in-game.
In Game Testing Procedures
This section is the most important and will detail the steps required for testing a mod in a consistent way so it can be recommended (or not recommended) for a Guide. It will be as simple and streamlined as possible, however, with the complexity, breadth, and depth that mods can be, this step will never be able to cover all mod testing scenarios. Below we provide the most common of scenarios, however, when the scenario isn't covered best-judgment must be used.
Screen & Video Captures
NOTEWhen screenshots or videos are needed for comparisons, provide them from in-game sources. Do not use "studio" applications to provide shots and/or video. In-game lighting conditions and rendering can change the appearance of many textures compared to studio applications. Therefore, it is very important to capture these comparisons from within Skyrim itself.
FRAPS is an good program for both screen and video captures in-game, as well as, capturing FPS data; however, feel free to use the preferred program for captures. The paid version of FRAPS provides more functionality and is recommended. PNG is the preferred format for screen captures due to its accuracy in capturing correct colors, saturation, tints, etc. JPEG is not recommended because some of the image's originality can be lost; however, it can be used if PNG isn't an option. Do not use GIF format for screen captures!
For video captures, AVI, MKV, and MP4 file formats (containers) are best. Using one of these three formats or a higher quality one for capturing video is very important for proper captures in high definition. Use the H.264 codec, if possible, and MPEG-4 as a second option. Video for true HD should be captured in at least 1080p. Audio for video captures, if it can be set, should be no less than a 48khz sample rate and no less than a 128kbps bit rate (96khz sample/384kbps bit rate is recommended for true HD audio). Use this information for encoding edited videos for compares as well. Adobe Premiere Elements is excellent for this, but rather expensive.
When uploading captures for posting compares on the forums, please use a 3rd party service to host captures. Do not store your captures on the STEP wiki! Imgur and Postimage have proven to be an excellent free image hosting sites for uploading screen captures to. Please use the editor functions on the forums to post compares. For posting video captures, please use YouTube. Other services have proven themselves annoying for community members to use; most requiring an account to view the videos.
Quick tip: Unless the monitor resolution is 1920x1080 or higher (and capturing at that resolution), do not encode your videos in 1080p. The result will be blurry due to upscaling. If below 1920x1080, 1600x900 for example, encode videos in 720p.
In-Game Mod Assessment
Enable the testing profile
Testing for Step Modification Guides should be done on a profile set up for that Guide in Mod Organizer.
Below are some recommended testing locations for Skyrim. Locations for other games should cover the majority of the games assets within them:
- QASmoke - Testing Hall (great for gathering needed items, but not great for screenshots or video due to lighting)
- Mod Specific Location
Testing Step 1 - The Vanilla Experience
It's important to be familiar with how the original content works whether it be a texture, a quest, a game mechanic, etc. Therefore, if the user is not familiar with the mod changes being tested, they should experience the original content first. This is also necessary for texture comparisons, which must have a vanilla shot for reference. To do this:
- Load the vanilla profile created earlier and launch the game.
- Fast travel or COC to the area closest to the mod's in-game location will be or find a good example of the assets being altered.
- Play the game for a period of time or make a save for comparison shots to experience or view the vanilla content.
- Close the game and review the findings for the next step.
Testing Step 2 - Experience the Mod
- Relaunch Mod Organizer and select the Step Modification Guide profile for further testing.
- Activate the mod being tested.
- Relaunch the game and load the vanilla save again or the save mod for comparison shots.
- Allow the mods to initialize.
- Play the game for a period of time assessing the mod or take your shot for the compare.
- Is it working as intended?
- Does it look as intended?
- Does the mod fit in with the game's content/ambiance/environment?
- Are there any issues?
- Save the game (new save) and reload it. Have any issues appeared?
- Close the game and review the findings.
Final Step - Review
- Gather the findings from all testing sources including the external testing.
- Summarize them into a review and post this review on the mod's topic on the forums. Things to include are, but not limited to:
- Issues with file structure and installation
- Conflicts with vanilla or Guide content (and solutions, if any)
- Whether the mod met or fell below expectations
- Any in-game issues
- Personal assessment of whether mod should or should not be included in the Step Modifications Guide
NOTEBe mindful to remain objective. Do not write anything that will reflect upon the mod's author in a personal manner. Only review the craftsmanship, reliability, operation, etc of the mod itself. We're assessing mods for Guide inclusion, not judging authors and their skills!
Denied Mods (optional)
At times there will be mods that do not make it pass evaluations for reasons other than Mandate violations or being less preferred over another mod. For these mods, consider contacting the mod's author with the findings so that they may improve their mod, or potentially provide solutions which will work for Guide inclusion. This is entirely optional. Some mod authors will appreciate the contact; however, others will not so use best judgement.
Capturing Texture Comparison Sets
When testing mesh/texture based mods, comparison screen shots should to be taken and posted for the community. These shots are not for the staff to use for evaluation, but rather for the community's benefit and our reference. These shots need to include a Vanilla shot, current Guide content shot, and a shot of the content of the mod being tested. To capture screen shots that work best for these comparisons use the best practices outlined below. This outline will allow the shots to be of the exact same screen, in each shot, with the only differences being the asset replacements.
- Active Immersive HUD, if not already, for all MO profiles used in the testing, including the vanilla profile. This will auto hide the HUD so that it doesn't have to be done manually from the console.
- Load the vanilla profile and vanilla save file. The mod being tested should not be active.
- While in-game, locate the vanilla texture(s) that need to be captured. Explore until the texture(s) are found or fast travel or COC to the texture's location, if known.
- Once the texture is found, line the shot up in the frame. Pay attention to lighting and angles to make sure the shot will be good for compares.
- Ensure there is nothing that will interfere in the shot such as an NPC walking into the frame.
- Save the game with a new save, if not loading from a previous save.
- Load the save and Do NOT touch that mouse!
- Once the game loads, make sure there is no text on the screen and Immersive HUD has hidden the HUD elements. When all is good, press the screen capture key for the program being used to capture.
- Now exit the game.
- Switch to the Guide profile.
- Repeat steps 7-9 above for the Guide shots.
- Now in the current Guide profile, active the mod being tested and ensure it is overwriting the desired assets.
- Repeat steps 7-9 to capture the shots from the mod being tested.
If completed correctly, a set of three compares should exist: a vanilla shot, a Guide shot, and a shot with the mod included. These can be used to post comparison shots on the forums. Simply repeat this process for all assets that require a comparison set for any given mod. When only one texture is being tested, it is nice to provide several compares from various angles/locations/lighting situations for a more complete comparison set.
Tools, Hints, and Utilities
Helpful Console Commands
Below is a list of helpful console commands to use while testing. Use these to expedite the testing while in-game. They are specific to Skyrim and other Bethesda game. To open the in-game console press the [ ~ ] (tilde) key, which is normally located just below the ESC key on a standard keyboard. Press it again to close the console.
- tmm 1
- This will toggle all map markers on; thus allowing you to fast travel to any location quickly. Enter 0 in place of 1, to reset to default.
- Toggles God Mode on/off making you invincible. Health, magicka and stamina will not run out either with enabled.
- Toggles collisions on/off. Don't use while falling or you may crash to desktop.
- Toggles Free Camera Mode on/off so you can fly around the environment. 'tfc 1' will pause the camera.
- Toggles Artificial Intelligence so characters will not react to the player.
- Toggle Combat Artificial Intelligence so NPCs will not attack the player.
- player.additem formID ###
- Adds the item to the player's inventory. Replace 'formID' with the item code of the item. Codes can be found here. Replace the '###' with the number of items to add. For example, to add gold: player.additem f 200 (this adds 200 gold to the player's inventory.)
Tip: leading zeroes can be dropped in the item code.
- player.addspell formID
- Adds a specified spell to the player's spell list. Spell codes can be found here.
- coc locationName
- coc qasmoke
- Transports the player to the developer testing hall.
- Unlocks the targeted object (doors, chests, etc). To target an item/object click on it while the console is open.
- Kills a targeted enemy.
- Kill all non-essential NPCs within the player's vicinity.
- Resurrects a dead target.
- set timescale to ##
- Changes the timescale of the game. 20 is the default setting. Setting this below 10 can cause issues.
Intel Texture Works Plugin - allows you to open, edit, save DDS files in Adobe Photoshop using the latest formats (including BC7).