Guide Talk:TES5Edit Mod Cleaning

Template:Notice by frihyland

Updated: 9:04:59 13 October 2012 (UTC)

GUIDE FORUM THREAD

Introduction

Section 1: Understanding Mod Cleaning

Section 2: Mod Cleaning Instructions

Glossary

Closing Remarks & Credits

[edit]

This guide is designed to introduce mod authors and mod users to the concept of mod cleaning and to provide instructions on how to clean mods using TES5Edit. Clean mods are essential to the creation of a stable modded game environment in which there are minimal conflicts between the mods used.

This guide is split into two sections: the first contains the information neccessary to understand why some mods need cleaning and the cleaning process itself, while the second provides the actual instructions for cleaning mods. There is also a Glossary at the end of the guide that may be used as a quick reference if you forget what a term means.

While this guide may seem intimidating, cleaning is not hard once you understand it. Devote some time to reading through this guide and you'll be well on your way to have a squeaky-clean mod list and a fun, more stable game environment.

[edit]

Why Clean?

Many mods make unintentional or unnecessary edits to game elements. Such edits can be a problem when using multiple mods due to the "rule of one", that states that if multiple mods change the same record, only the changes from the mod loaded last will be applied.

When a mod containing unintentional or unneccessary edits (known as a "dirty mod") is used with a mod that makes an intentional edit to one or more of the same elements, this can introduce the possibility of bad things happening, from incorrect game settings to broken content and crashing to desktop (CTD). The more dirty mods you run, the more likely you are to encounter such issues.

Dirty Edits

Dirty edits occur for two reasons: the first is that a mod author changes a record's properties but then decides to undo their changes by setting the properties back to their original values instead of removing the changes entirely, and the second is due to Creation Kit bugs changing the values of properties without the mod author being aware of this occurring.

The most common type of dirty edit is an Identical To Master (ITM) edit. This is where a mod contains an edit to a record where the edited values are identical to the original values. There are other types of dirty edit, including things like automatic cell water height edits, but they are commonly also referred to as ITM edits, so the terms "dirty edit" and "ITM edit" are somewhat colloquially equivalent.

Example 1: Identical To Master Edits

Consider a quest mod in which the author has also changed the properties of some sneaking settings to better fit with other areas of the quest mod. The author then later decides the sneaking changes are unnecessary, and sets them back to their original values. If this quest mod were then loaded after a mod that overhauls the sneaking system, the sneaking overhaul's effects would be at least partially undone by the quest mod, which is not the effect intended by the author of either mod, nor the effect desired by the user.

The solution would be to clean the quest mod, which would remove the ITM edits and allow the sneaking overhaul mod to function as intended.

Wild Edits

Wild edits are unintentional or misdirected edits to game elements by a mod author that cause unneccessary conflicts with other mods. They are called wild edits because they often appear to have been made hapazardly with no relation to the author's stated intentions.

One example of a wild edit would be when a container is incorrectly modified to include an item, but instead of only that one container including the item, the result is that all containers of the same type include the item. If the description of the mod states that an item is added to a specific container, then it is clear this is a wild edit and the mod should be cleaned by fixing the wild edit.

Deleted References

Deleted references are a common cause of crash-on-exit. This is because if one mod tries to modify a reference that another mod has deleted, then the first mod cannot find the reference, and this then leads to a crash when the game is exited.

References are generally deleted to remove them from the game world, but a practically identical effect can be achieved by undeleting the reference then disabling it and moving it to a position where it is no longer visible in the Creation Kit (TES5Edit sets the Z axis position to -30,000 units).

What Is Cleaning?

Cleaning very broadly defined is a process that analyzes all the changes a mod makes and determines which edits are actually supposed to be in the mod, then removes those that are not.

Cleaning can be automated when dealing with dirty edits and deleted references, but must be performed manually when dealing with wild edits. This is because while dirty edits and deleted references have clear technical definitions, identification of wild edits requires knowledge of the mod author's intent, which is not really possible for a computer to determine accurately or consistently.

Manual Cleaning

Manual cleaning is a fairly advanced technique that requires those performing it without the aid of specific step-by-step instructions to know how TES5Edit works and to know a bit about how mods work. Mod users will generally not need to perform manual cleaning most of the time, and so the specifics are not covered in this guide.

Manual cleaning is all about using your judgement to decide whether an edit is a valid, intentional edit or a wild edit that should be removed. Wild edits may be obvious such as in the example below, or not obvious, such as some rocks that were accidently moved a small amount while editing a cell. Once you have judged that an edit is wild, you must then use your knowledge of modding to decide on the correct method of fixing it, and perform the fix correctly. Incorrectly judging an edit to be wild, choosing the wrong method of fixing a wild edit and performing the fix incorrectly will all lead to making the situation worse, which is why knowledge and experience are required when manually cleaning plugins.

Example 2: Manually Cleaning A Wild Edit

Consider a mod that is intended to add a powerful sword named the Super Sword of Slaying to the bottom of a new dungeon. However, what it instead does is to change the Silver Sword so that it is now the Super Sword of Slaying, and adds one to the bottom of a new dungeon. The result is that every Silver Sword in the game will now be a Super Sword of Slaying.

To fix this wild edit, the FormID of the edited Silver Sword record in the mod plugin can be changed from 00###### to 01######, which will make it a new type of weapon instead of an edited Silver Sword. The reference placed in the dungeon can then be updated to use the new FormID. This will result in a Super Sword of Slaying being present in the dungeon, while all Silver Swords remain Silver Swords.

If you don't understand the above example and the fix, you don't possess the knowledge required for manual cleaning, so don't worry about it and just stick to automated cleaning.

Automated Cleaning

Automated cleaning requires knowledge of exactly which other mods the mod to be cleaned is dependent on, both implicitly and explicitly.

In automated cleaning, if cleaning dirty edits, the program analyses every record edit a mod makes and compares each one against the record without the edit, removing any edits that match the technical definitions of dirty edits.

If cleaning deleted references, the program it analyses each reference a mod edits, checking if it has been deleted or not, undeleting and disabling any deleted references encountered.

How To Clean?

Even when performing automated cleaning, it is important to ensure everything is set up so that cleaning can be performed correctly.

Order Of Cleaning

It is recommended that you begin cleaning at the top of your load order and just do down the list. This will ensure that dependencies will be cleaned before the mods that are dependent on them, so long as yout load order is correct.

It is important to ensure a mod's dependencies are clean before cleaning the mod itself because the automated cleaning process will assume that the dependencies are clean. If this assumption is incorrect, it may lead to the mod being incorrectly cleaned.

Loading Plugins Correctly

The terms explicit dependency and implicit dependency are used in this section. It is important that you understand these terms, so before continuing, here are their definitions:

An explicit dependency is a plugin that another plugin relies on, and which has its filename explicitly referenced within the latter plugin. Explicit dependencies are easily determined because they can be viewed in Wrye Bash, and if an explicit dependency is missing or loaded after the plugin that depends on it, the game will crash on startup. A plugin's explicit dependencies are also known as its masters.

An implicit dependency is a plugin that another plugin depends on, but which is not referenced by the latter plugin. Implicit dependencies must be determined by checking if the mod author has listed any required mods for their mod, and by considering the nature of the mod. For example, compatibility patch plugins are often implicitly dependent on the plugins of the mods they resolve compatibility issues for.

Having the correct load order set for your mod plugins is essential for them to be cleaned correctly. Load ordering is outside the scope of this guide, but the easiest way to get a load order that is likely to be correct is to use BOSS.

It is highly recommended that when cleaning a plugin, only the plugin itself and the plugins it depends on should be loaded. Having any other plugins loaded introduces the risk of their contents interfering with the cleaning process and leading to invalid results.

There are two absolute requirements to loading plugins for cleaning:

1. The plugin to be cleaned must be the last loaded plugin in your load order.
2. Any plugins upon which the plugin to be cleaned depends must be loaded. This includes both implicit and explicit dependencies.

Due to these rules, you cannot clean more than one mod in the same TES5Edit session. If you have multiple plugins to clean, then they must each be loaded into separate TES5Edit sessions and cleaned separately.

Example 3: Loading Explicit And Implicit Dependencies

Consider a plugin "VillageMod-QuestMod Patch.esp" that acts as a compatibility patch between VillageMod.esp and QuestMod.esp.

VillageMod.esp changes (among other things) some of the names of the NPCs in the village and changes the landscaping around it, but QuestMod.esp includes voice-acted dialogue that uses the original NPC names, and adds a wall around the village that is partially buried by the landscape changes of VillageMod.esp when the two are used together.

"VillageMod-QuestMod Patch.esp" reverts the names of the NPCs back to their originals to match with QuestMod.esp's voice acting and moves the walls added by QuestMod.esp so that they are no longer buried. As it changes the NPC names, it has Skyrim.esm as an explicit dependency, and as it changes the walls, it has QuestMod.esp as an explicit dependency. However, because it doesn't change anything added by VillageMod.esp, VillageMod.esp is an implicit dependency.

"VillageMod-QuestMod Patch.esp" will refuse to load without also loading Skyrim.esm and QuestMod.esp as they are explicit dependencies, but it is possible to load it without also loading VillageMod.esp. However, if this is done then the name changes in "VillageMod-QuestMod Patch.esp" will appear to be Identical To Master edits, as they will be identical to the names in Skyrim.esm and QuestMod.esp does not touch them, and so the name changes in the patch will be incorrectly removed. If all three plugins are loaded, then the name changes will not appear to be Identical To Master edits, as they are different to the names in VillageMod.esp.

A Community Effort

The principle reason for the presence of dirty mods is a lack of awareness of the issues surrounding dirty mods and the importance of mod cleaning. As such, a community-wide effort is required to raise awareness and so reduce the incidence of dirty mods.

It's NOT a good idea for everyone to just go ahead and clean their full load order and then forget about it.

The problem is twofold. First, to properly clean mods usually requires an understanding of the intent of the mod author. There might be implicit dependencies on other mods which are not part of the master list. Second, and following from the first, depending on how the cleaning was done, people end up with slightly different versions of these modules, which is going to make support in case of problems a nightmare for the mod author.

Mod quality is a community-wide problem that needs to be properly addressed on a community level. Mod authors need to ensure that their mods are clean, because they only need to clean their mod once for all future users to benefit, whereas each and every user of a dirty mod would have to clean it themselves, which is hugely inefficient.

The BOSS team provides a focus for community efforts to raise awareness regarding mod cleaning, by accepting reports of dirty mods and attaching them to the relevant plugins in the BOSS masterlist as messages containing a link to this wiki page, along with any additional information supplied. This data is then extracted and displayed to the user by BOSS when it runs, and the data can also be accessed by other utilities that make use of the BOSS API. The result is a central repository of information that can be accessed by a large number of people.

For mod authors:

1. Clean your mods properly. All of them. You are the one who really knows the intention of your mod. Closely review all changes to make sure they preserve the intent of your mod.
2. Cleaning mods is NOT just limited to running the automated functions and be done with it. Manually review every override record your module(s) contain to make sure that that's really what you intended to do.
3. List in your documentation that you HAVE cleaned your mods and which version of TES5Edit you used for it. If there were any implicit dependencies that you had to observe while doing the cleaning, specifically list them and the rational for them.

For mod users:

1. Check any mods you use that do not specifically mention being properly cleaned (by basically doing a dry run of the cleaning process).
2. If any of your mods are dirty, inform the author. The mod authors are the ones that really understand the intention of their mods. They are the ones that are in the best position to properly clean them. And if they only publish properly cleaned mods, the whole community benefits from it.
3. If a mod has been deserted by its author or the author is unwilling to fix his mess, just leave a short note on the comments and/or in the RELz thread to save other mod users from wasting their time. Remember: stay civil; flaming doesn't help anyone.
4. If any of your mods are dirty, submit them to the BOSS team as directed in the BOSS readme, so that others can benefit from your findings.

[edit]

These are the actual instructions for the automated cleaning of plugins using TES5Edit:

1. Download TES5Edit.
2. Extract the downloaded archive to wherever you want.
3. Run the TES5Edit.exe file.
4. A dialog box appears listing all your installed plugins, with all currently active plugins ticked. Right-click anywhere in the list and pick "Select None".
5. Tick the boxes for the plugin you want to clean and for any of its implicit dependencies. You don't need to tick the boxes for its explicit dependencies as TES5Edit will automatically load them anyway.
6. Click "OK" in the dialog box.
7. A mod group dialog box may be displayed. Never tick any of the boxes in this dialog box when cleaning plugins. Click "OK" in the dialog box.
8. After the plugins have finished loading, right click the plugin you want to clean and select "Apply Filter".
9. A large dialog box with many options will be displayed. Make sure all the checkboxes are unticked apart from the "Conflict status inherited by parent" checkbox, which should be ticked.
10. Click "OK" in the dialog box. Filtering will likely take a long time.
11. Right click on the plugin you are cleaning and pick “Remove "Identical to Master" records”.
12. Wait two seconds when the big warning message appears and click “Yes…”.
13. The plugin name will turn bold if any dirty edits were found and removed. Check the Messages tab to see what was changed. Make a note of the number of ITM records removed for when reporting the plugin to the BOSS team.
14. Right click again on the plugin you are cleaning and select “Undelete and Disable References.” Again, check the Messages tab to see what was changed. Make a note of the number of undeleted and disabled references for when reporting the plugin to the BOSS team.
15. Try to close the program. If there were no dirty edits or deleted references found, the program will close immediately. Otherwise, a dialog box will be displayed, listing the plugin you cleaned.
16. Make sure that the checkbox for the plugin you cleaned is ticked, then click "OK". This will save the cleaned plugin. A backup of the plugin before cleaning is placed in your Data folder, with its file extension being the timestamp at which it was created.

[edit]

This glossary contains definitions for the general modding terms and cleaning-specific terms used in this guide.

Mod: A modification for Skyrim. Can be player-made or official DLC.

Plugin: A file with a .esp or .esm file extension (well, it's what's inside the file that matters, but the file extension is a pretty good clue).

Record: A data structure inside a plugin that holds the information on one element of the game. For example, each type of weapon, each race, each potion, etc. have different records.

Rule of One: Only one plugin's changes to a record can be applied by the game. If multiple plugins change the same record, then the changes from the last of those plugins in the load order will be applied. There are a few exemptions, eg. if multiple plugins put things into the same cell, then all those things will be seen in-game, but in general the rule holds for non-complex record types (and complex record types are beyond the scope of this guide).

Load order: The order in which plugins are loaded.

Dirty mod: A mod that contains dirty edits, wild edits, deleted references or any combination of them.

Dirty edit: An edit made which is unnecessary, often caused by Creation Kit bugs or authors making unintended edits then undoing them incorrectly. Such edits can be cleaned automatically.

Identical To Master edit: A type of dirty edit made where the edited value(s) is/are identical to the value(s) without the edit.

Wild edit: An edit made which is not consistent with the intent of the mod, often caused by authors accidently making small edits or trying to perform an edit in the wrong way. Such edits must be cleaned manually.

Deleted reference: A reference to a record that has been deleted. For example, placing a Silver Sword into the game world creates a Silver Sword reference there, which may then be removed by deleting it.

Dependency: A plugin upon which another plugin relies. For example, Skyrim.esm is a dependency for most mods.

Explicit dependency: A dependency that is recorded within the dependent plugin, such that the game cannot be loaded without the dependency being satisfied.

Implicit dependency: A dependency for which there is no indication given within the dependent plugin, and which can only be determined by considering author intent.

[edit]

This guide has been adapted from the TES4Edit Cleaning Guide. Much of the content of that guide has been removed for this guide, with all lists of mods being absent: this is partly because there are as yet no mods to list, but mainly because such lists belong on a separate wiki page from the guide. The special cleaning instructions section is also missing for the same reason, and the tips on avoiding dirty mods section is missing because I felt it was counter to the topic of the guide. I also left out the review of key concepts because I felt that having such a section would cause many readers to skip the rest of the information, and so put them in a position of false confidence from where they could do significant harm to their modded game.

I have also created a [Dirty_Plugins_List|Dirty Plugins List] page for the listing of any plugins that need cleaning, plugins that should not be cleaned, and plugins for which manual cleaning instructions are required.

Many thanks go to the various authors of and contributors to that article, including but not limited to: Brumbek, Arthmoor, Vorians, ElminsterEU and dev_akm.

Thanks also go to ElminsterEU, Sharlikran, Zilav and Leandro Conde for their work in getting TES5Edit released.

Finally, thanks go to the BOSS team members responsible for Skyrim's masterlist, for their efforts in collating dirty mod reports.

Still to do:

• Include pictures for the instructions a la Mod Cleaning Tutorial.
• Get the correctness of the guide verified by the TES5Edit team.
• Create a separate wiki page to hold the list of mods needing cleaning (and which should not be cleaned, if there will be any), and instructions on how to generate the list from BOSS's masterlist.