Mutagen / Synthesis Test Drive

[Mousetick]

Mutagen is an object-oriented framework for parsing and generating Bethesda game plugins. Synthesis patchers are programs built on top of the Mutagen framework for a specific purpose. Both are written in the C# language. Similarly to zEdit or xEdit patcher scripts, they allow for analyzing an entire load order worth of plugins and producing a specific patch algorithmically.

Synthesis patchers come in 2 forms: standalone .exe executable, or source code.

---

I first tried one of the standalone patchers distributed on Nexus but it auto-detects the game Data folder location based on the Windows registry setting, while my modded SSE installation is stored in a separate folder in a non-standard location not managed by Steam.

The patcher auto-detected and ran on my unmodded SAE installation. So that experiment didn't go very far and I can't speak of its results, which were nil. When run from MO, the standalone patcher runs quickly in a console window that closes immediately upon completion, making it difficult to see what went on.

Using a standalone patcher may be fine if a single one is needed, however using the hub GUI should be preferable for combining multiple patchers, reducing the number of plugins generated and benefitting from the "pipelining" mechanism.

---

I then switched to the hub GUI.

• It requires the .NET 6 SDK in order to compile source code patchers into executable form. The SDK needs to be downloaded and installed separately, as it's a developer tool not built into Windows.
• The UI is pretty nice and fairly easy to use, at least compared to other modding tools. It suffers from "kool flat UI" syndrome: one has to hover everything in order to find out if it's a button or not.
• Different game profiles (e.g. Skyrim SE, Enderal SE, Fallout 4) can be created and activated, including ones in custom folder locations.
• The list of available patchers is pulled from a central registry on GitHub. Each individual patcher is downloaded in source code form from its own GitHub repository. There are quite a few patchers, but not many. I expected there'd be more of them given there are thousands of Skyrim SE mods.
• Most patchers are very "niche" being specialized for one specific mod, or one specific type of change.
• Multiple patchers can be run in a "pipeline" feeding from one to the next and producing one final combined patch plugin.
• Some patchers are configurable, either via a settings panel integrated in the hub GUI, or via a configuration file to be edited in a text editor.

---

I tried a few patchers, some that were relevant to my modlist, others simply out of curiosity:

1. AI-Overhaul-Patcher^*: Forwards NPC changes from AIO and merges them with NPC changes from other mods. As LOOT sorts AIO pretty high in the load order, its NPC changes are likely lost by other mods loaded after it. The patcher seems to work well enough as designed, but I don't agree or don't understand why it doesn't forward some AIO edits. Overall I can do the same or better job manually in xEdit. Admittedly there are only a handful NPCs in STEP that need patching (because of Simple Children and CRF) so it's easy to do manually. The automated patcher may be more useful to players who for example use mods changing the appearance of lots of NPCs.
2. AOSISCSoundPatcher: Supposed to add sound effects to all weapons & armor of mods in the load order, consistent with the changes in Audio Overhaul Skyrim or Immersive Sound Compendium. This produces absolutely nothing for me, even after disabling all my AOS compatibility patches. I don't really know what to expect - this is one of those patchers with zero documentation.
3. BreakdownRecipeGenerator: Generate breakdown recipes for all craftable items in a load order. Also provides the option of creating custom breakdown recipes for miscellaneous items, weapons and armors. This seems to work quite well, it generated breakdown recipes for seemingly every object. The only problem is I have no idea why I would need them or what I would do with them.
4. enblightpatcher: Tweak all light sources for ENB Light compatibility. This is a pretty extensive patcher that modifies the lighting properties of certain cells and of all light sources, of all mods in the load order, making them less bright. It's useless to me as I don't use ENB, but it's a good example of what can be done efficiently and reliably. It beats having to edit every single candle or torch in the game manually in xEdit.
5. ExperienceMutagenPatcher: Generates an Experience XP configuration file for killing races or NPCs added by mods in the load order. Appears to be functional. I intend to use the other patcher described further down rather than this one.
6. facefixer: Supposed to patch NPCs to fix the 'black face' bug. It definitely does something, but in the STEP modlist, the only mod that changes NPC appearance is Simple Children and there is no black face issue, so patching is unnecessary. Probably useful to players using a lot of NPC beautification mods.
7. leveledlistresolver: Merges levelled lists, like Wrye Bash, without relying on Bash tags. Appears to work quite well and to be pretty smart. I have to manually merge levelled lists in xEdit because Wry Bash produces a complete mess whenever I use it. I was quite happy to see this patcher produced results identical to mine, even better, it caught a list I missed.
8. Realistic-Water-Two-Patcher: Forwards changes from RWT so they don't get overwritten by other mods. Similar to AI-Overhaul-Patcher above, except this one is dumb and rewrites all RWT-modified records, whether they are overwritten or not, producing a huge patch.
9. Requiem-Experience: Generates an Experience XP configuration file and patch plugin, to be used along with Requiem - or any other overhaul which makes Skyrim world to be un-levelled. This is a more feature-rich patcher than the ExperienceMutagenPatcher above. Good documentation and customization options. I only used a subset of its options with a simple vanilla-like setup (no Requiem, no unlevelling overhaul), but it produced useful results.

As with any automated tool like Mator Smash or Wrye Bash, it's a good idea to review the generated patches in xEdit, at least cursorily, to verify the results look ok.

---

Overall, I'm not too impressed but there's definitely some potential and I found some useful patchers on offer, which I'm going to rely on in the future. Quality appears to vary greatly, the most disappointing aspect for me being the total lack of documentation for many patchers. It's all open-source, so sure, it's possible to read the source code to figure out how a given patcher works, but not everyone can read and reverse-engineer C#, or spend the time to do it.

On the plus side, the patching process is pretty easy to set up. Once set up, the whole patching pipeline can be run again from MO whenever a mod is added or removed, and it's quite fast (compared to manual editing in xEdit). In theory it can support huge load orders.

On the minus side, the fact that patchers are actual programs, written C#, represents a high obstacle to making a patcher.

The Mutagen/Synthesis toolset could be quite useful to automatically generate configuration files for SPID, Base Object Swapper, Keyword Item Distributor, and similar framework mods, based on all mods in a given load order.

---

(*) Before actually trying out the AI Overhaul patcher, I wildly imagined that it would be capable of applying AIO changes to new NPCs from other mods. I was completely wrong and in retrospect, I realize this would be impossible. How could a particular NPC behavior be assigned to an unknown NPC from an unknown mod? Perhaps via the use of specific keywords to 'tag' mod-added NPCs. That might work. In any case, the AIO patcher doesn't work like that at all, it only deals with NPCs that AIO supports 'out-of-the-box' in its plugin.

[z929669]

Thanks for the write-up. It sounds like Mutagen/Synthesis may be the way to go if/when patchers evolve for mods of interest (Requiem for me). I like the ELE and ENB Light patcher concepts. Need one for TCIY and CACO!

Could get really interesting as it evolves.

[z929669]

On 1/16/2022 at 11:45 AM, Mousetick said:

Mutagen is an object-oriented framework for parsing and generating Bethesda game plugins. Synthesis patchers are programs built on top of the Mutagen framework for a specific purpose. Both are written in the C# language. Similarly to zEdit or xEdit patcher scripts, they allow for analyzing an entire load order worth of plugins and producing a specific patch algorithmically.

Synthesis patchers come in 2 forms: standalone .exe executable, or source code.

---

I first tried one of the standalone patchers distributed on Nexus but it auto-detects the game Data folder location based on the Windows registry setting, while my modded SSE installation is stored in a separate folder in a non-standard location not managed by Steam.

The patcher auto-detected and ran on my unmodded SAE installation. So that experiment didn't go very far and I can't speak of its results, which were nil. When run from MO, the standalone patcher runs quickly in a console window that closes immediately upon completion, making it difficult to see what went on.

Using a standalone patcher may be fine if a single one is needed, however using the hub GUI should be preferable for combining multiple patchers, reducing the number of plugins generated and benefitting from the "pipelining" mechanism.

---

I then switched to the hub GUI.

• It requires the .NET 6 SDK in order to compile source code patchers into executable form. The SDK needs to be downloaded and installed separately, as it's a developer tool not built into Windows.
• The UI is pretty nice and fairly easy to use, at least compared to other modding tools. It suffers from "kool flat UI" syndrome: one has to hover everything in order to find out if it's a button or not.
• Different game profiles (e.g. Skyrim SE, Enderal SE, Fallout 4) can be created and activated, including ones in custom folder locations.
• The list of available patchers is pulled from a central registry on GitHub. Each individual patcher is downloaded in source code form from its own GitHub repository. There are quite a few patchers, but not many. I expected there'd be more of them given there are thousands of Skyrim SE mods.
• Most patchers are very "niche" being specialized for one specific mod, or one specific type of change.
• Multiple patchers can be run in a "pipeline" feeding from one to the next and producing one final combined patch plugin.
• Some patchers are configurable, either via a settings panel integrated in the hub GUI, or via a configuration file to be edited in a text editor.

---

I tried a few patchers, some that were relevant to my modlist, others simply out of curiosity:

1. AI-Overhaul-Patcher^*: Forwards NPC changes from AIO and merges them with NPC changes from other mods. As LOOT sorts AIO pretty high in the load order, its NPC changes are likely lost by other mods loaded after it. The patcher seems to work well enough as designed, but I don't agree or don't understand why it doesn't forward some AIO edits. Overall I can do the same or better job manually in xEdit. Admittedly there are only a handful NPCs in STEP that need patching (because of Simple Children and CRF) so it's easy to do manually. The automated patcher may be more useful to players who for example use mods changing the appearance of lots of NPCs.
2. AOSISCSoundPatcher: Supposed to add sound effects to all weapons & armor of mods in the load order, consistent with the changes in Audio Overhaul Skyrim or Immersive Sound Compendium. This produces absolutely nothing for me, even after disabling all my AOS compatibility patches. I don't really know what to expect - this is one of those patchers with zero documentation.
3. BreakdownRecipeGenerator: Generate breakdown recipes for all craftable items in a load order. Also provides the option of creating custom breakdown recipes for miscellaneous items, weapons and armors. This seems to work quite well, it generated breakdown recipes for seemingly every object. The only problem is I have no idea why I would need them or what I would do with them.
4. enblightpatcher: Tweak all light sources for ENB Light compatibility. This is a pretty extensive patcher that modifies the lighting properties of certain cells and of all light sources, of all mods in the load order, making them less bright. It's useless to me as I don't use ENB, but it's a good example of what can be done efficiently and reliably. It beats having to edit every single candle or torch in the game manually in xEdit.
5. ExperienceMutagenPatcher: Generates an Experience XP configuration file for killing races or NPCs added by mods in the load order. Appears to be functional. I intend to use the other patcher described further down rather than this one.
6. facefixer: Supposed to patch NPCs to fix the 'black face' bug. It definitely does something, but in the STEP modlist, the only mod that changes NPC appearance is Simple Children and there is no black face issue, so patching is unnecessary. Probably useful to players using a lot of NPC beautification mods.
7. leveledlistresolver: Merges levelled lists, like Wrye Bash, without relying on Bash tags. Appears to work quite well and to be pretty smart. I have to manually merge levelled lists in xEdit because Wry Bash produces a complete mess whenever I use it. I was quite happy to see this patcher produced results identical to mine, even better, it caught a list I missed.
8. Realistic-Water-Two-Patcher: Forwards changes from RWT so they don't get overwritten by other mods. Similar to AI-Overhaul-Patcher above, except this one is dumb and rewrites all RWT-modified records, whether they are overwritten or not, producing a huge patch.
9. Requiem-Experience: Generates an Experience XP configuration file and patch plugin, to be used along with Requiem - or any other overhaul which makes Skyrim world to be un-levelled. This is a more feature-rich patcher than the ExperienceMutagenPatcher above. Good documentation and customization options. I only used a subset of its options with a simple vanilla-like setup (no Requiem, no unlevelling overhaul), but it produced useful results.

As with any automated tool like Mator Smash or Wrye Bash, it's a good idea to review the generated patches in xEdit, at least cursorily, to verify the results look ok.

---

Overall, I'm not too impressed but there's definitely some potential and I found some useful patchers on offer, which I'm going to rely on in the future. Quality appears to vary greatly, the most disappointing aspect for me being the total lack of documentation for many patchers. It's all open-source, so sure, it's possible to read the source code to figure out how a given patcher works, but not everyone can read and reverse-engineer C#, or spend the time to do it.

On the plus side, the patching process is pretty easy to set up. Once set up, the whole patching pipeline can be run again from MO whenever a mod is added or removed, and it's quite fast (compared to manual editing in xEdit). In theory it can support huge load orders.

On the minus side, the fact that patchers are actual programs, written C#, represents a high obstacle to making a patcher.

The Mutagen/Synthesis toolset could be quite useful to automatically generate configuration files for SPID, Base Object Swapper, Keyword Item Distributor, and similar framework mods, based on all mods in a given load order.

---

(*) Before actually trying out the AI Overhaul patcher, I wildly imagined that it would be capable of applying AIO changes to new NPCs from other mods. I was completely wrong and in retrospect, I realize this would be impossible. How could a particular NPC behavior be assigned to an unknown NPC from an unknown mod? Perhaps via the use of specific keywords to 'tag' mod-added NPCs. That might work. In any case, the AIO patcher doesn't work like that at all, it only deals with NPCs that AIO supports 'out-of-the-box' in its plugin.

I did an experiment to test some of my own LL patching against that of Synthesis (leveledlistresolver) and WB (bashed patch, only selected LL). They perform similarly but not exactly the same. What I did was disable all of the Step patches (WIP dev versions for 2.2.0 build under the latest game runtime), sorted with LOOT, and ran each patcher independent of the other against the identical LO (minus the WIP Step patches). Then I loaded it all in xEdit with the Step patches active to spot check. Here's some examples for different types of loot:

I haven't dug much into the results yet, but here's some of my initial take-aways:

• Synthesis only seems to touch weapons, armor, and clothing-related (and restoration-related) LLs (like gems), while WB considers 30-50% more LLs. I presume this to be due to it's use of bash tags (and all relevant mods have bashed tags ... see below). Synthesis doesn't use them and may instead use a narrower scope.
• Neither WB or Synthesis picked up on the AMB gauntlets, presumably, because they are not included in the conflicting LLs. Same goes for the CACO ingredients related to health/magica. That's a false statement and a reflection of my not having examined the output closely enough. Both automated methods consider the AMB LL and propose sensible treatments.
• Synthesis seems to exclude vanilla master LL entries, while WB tends to include them.
• Both patcher serve as a good basis for reference to ensure all/most relevant LLs are addressed, but the algorithms seem very basic for both automated patchers (i.e., each really seems to consider conflicting LLs rather than what's available in other mods).

Relevant mods are WACCP, AMB Content Addon, CCOR, CACO, USSEP, and CC content (for the most part anyway).

 

 

 

[z929669]

After a bit more investigation, I've determined the following for the use case example shown in the image (previous post corrected to reflect):

Both Synthesis and WB correctly merge the LLs with respect to AMB iron gauntlets. The Step patch directly adds the AMB engraved iron gauntlets (#1 above; ArmorIronGauntletsEngraved_AMB "Iron Gauntlets" [ARMO:2D06F984]), but it's probably best to utilize the sublist (#2 above; LItemArmorGauntletsIronEngravedVariants_AMB [LVLI:2D370C67]), which also includes the AMB engraved iron gauntlets in the intended ratio set by AMB Content Addon:

Additionally, WB (along with CC FearsomeFists) forwards the two vanilla LLs, but AMB does not intend this ... so I think Synthesis handles this in a way that preserves the intent. Therefore, I will be modifying the Step patch LL to reflect the Synthesis result in this case (and I'll be examining all LLs similarly using both WB and Synthesis treatments as examples).

My conclusion is that these automated-patchers provide a good basis for patching LLs (and probably many other record types), because they are comprehensively and consistently applying their respective algorithms, which I can't say is true when it's done manually. Manually sorting through all of the changes made by the automation (let's think of them as 'proposed' changes) allows fine tuning and corrections without missing anything ... speaking for myself, anyway.

EDIT: Just for kicks, the following is the Synthesis(leveledlistresolver) output for the record in question:

LItemArmorGauntletsHeavySpecial [1031C6:Skyrim.esm]
Skyrim.esm -> aMidianBorn_ContentAddon.esp
Skyrim.esm -> CC-FearsomeFists_CCOR_Patch.esp