[WIP] Mator Smash

[Mator]

 

Mator Smash

VIDEO TUTORIAL - For v0.0.7

GamerPoets Video - For v0.4.1

 

 

Mator Smash is now available on Nexus Mods. 

Nexus Mods Mod Page

 

 

As of v1.0.0, Mator Smash has a "Quick Patch" button

As of v0.5.1, Mator Smash will copy records with "errors" in them (such as unresolved/unexpected references)

As of v0.5.0, Mator Smash has new algorithm features and is more stable.

As of v0.4.1, Mator Smash merges redundant plugins properly.

As of v0.4, Mator Smash now produces complete record prototypes

As of v0.3, Mator Smash supports Fallout 4 and Skyrim SE

As of v0.2.2, Mator Smash allows you to modify tags on plugins from within the program.

As of v0.2.1, Mator Smash has Smash Settings for Skyrim for every Bash Tag Wrye Bash offered for Oblivion.

 

Changelog:

 

 

v1.0.0

- No longer cleaning masters (the smashed patch will require all patched plugins as masters)
- Increased saturation of profile panel colors
- Removed unimplemented buttons from the QuickBar
- Removed MO Integration - it's unnecessary
- Added easter egg
- Added "quick patch" button
- Fixed FO3/FNV load order issue
- Added Fallout3 and FalloutNV Smash.All settings  
- Updated FO4 and TES4 Smash.All settings
- Fixed switch smash profile button

 

v0.5.1

- Fixed inactive plugins being enabled for SSE/FO4
- Updated Smash.All so Perk Effect Conditions are treat as single entity
- Fixed case sensitivity when determining if patch destination directory is the game's data path
- Fixed bug with not creating directories when building a patch when user is not using MO
- Now allowing copying of records with "errors" in them
- Improved handling of out of order subrecords

 

v0.5.0
- Added override deletions algorithm feature

- Added force value algorithm feature

- Fixed some SSE/FO4 load order issues

- Added and updated smash settings

- Fixed issue with CTDs due to counter elements being set to 0 when their associated array is not present.

- Fixed issue with cells being deleted incorrectly due to bad ITM/ITPO cleanup code.

 

v0.4.1
- Adjusted the description of the Smash.All setting.
- Fixed merging of redundant plugins.
- Fixed recognition of sorted subrecord arrays.

v0.4

- Added Smash.All setting, a new version of the old "Automatic" setting.
- Dramatically improved record prototyping.  Record prototypes should now be complete and include unions such as BOD2 and BODT.
- Fixed loading the Skyrim load order file for SSE when not using MO.
- Fixed SSE load order DLC positions.
- Added Mod Organizer instancing support.

 

v0.3.1

- Fixed bug with initialization exceptions not being handled properly
- Fixed Fallout 4 BA2 file loading

 

v0.3
- Fixed bug with enabling automatic updating causing catastrophic failure.
- Added some language strings for "Enable", "Disable", and "Autoscroll"
- Fixed bug with setting corruption when combining settings (wasn't cloning the objects properly)
- Fixed bug with being unable to delete patches with unloaded plugins
- Fixed a bug with deleting patches
- Added "Remove unloaded plugins" context menu option to Smashed patches
- Now loading BSAs on initialization so localization strings are available to the program.
- Fixed cancelling plugin selection not closing the program
- Fixed floating point division by zero error when smashing records
- Added search functionality to the records tree
- Added SSE game mode support
- Added FO4 game mode support
- Adjusted how the profile form works.  Now it will always create default profiles for games that don't have profiles.
- Disabled the registration controls on the options form because they aren't working yet (we don't have a backend yet, after all)

 

v0.2.2.2

- Fixed bug with application being unable to create the profiles folder.
- Fixed index out of bounds when trying to set a plugin's smash setting.

 

v0.2.2
- Refactored project to use decoupled architecture similar to Merge Plugins
- Added a Plugin Selection Window
- Cleaned up the Tree Popup Menu, similar items have been grouped
- Added Tag Manager Form, which allows you to manually clear tags from the plugin, remove specific tags from the plugin, add tags to the plugin, or edit the plugin's description manually.
- If the "Create combined setting" checkbox is ticked when you close the Tag Manager, it will create and apply a virtual combined setting to the plugin for the tags you selected.
- Added a tags submenu for plugins, you can now clear tags on plugins, apply tags to a plugin based on your smash setting, or manage tags from the Tag Manager.
- Fixed various access violations that could sometimes occur when closing the program or when smashing certain records
- Added a feature that tells smash to merge mods that are comprised entirely of override records.  You can enable it from the Patching tab of the Options Window.  You will be notified in the smashing log of any redundant plugins that can be disabled after building your smashed patch.
- Fixed a bug with parsing ungrouped tags on plugins
- Fixed a bug with renaming a setting not updating its name in the Smash Settings List View in the Settings Manager.
- No longer creating duplicate virtual combined settings.
- Fixed bug where resizing the window didn't always update the contents of the form.
- Added a setting which can be manually added to the settings.ini file if the for simple splash display.  This will disable the glow on text in the Splash Form.  If the text on the Splash form looks distorted you should enable this setting.
- Now trimming leading and trailing whitespace when parsing tags.  So the following: `{{BASH: Delev, Relev   }}` will be parsed the same as `{{BASH:Delev,Relev}}`
- Added submenus for smash setting groups in the Plugins Popup Menu -> Add Smash Setting submenu
- Now verifying patch name and filename don't have illegal characters that aren't allowed by Windows.

 

v0.2.1

- Fixed access violation due to a flaw in the for loop when removing ITPOs
- No longer allowing building of patches which have one plugin.  This was causing a floating point division by zero error when performing preliminary calculations for the progress bar.
- Now disables the name, color, description, and tree controls when the msSettingManager is first created
- Now using a release build configuration
- Plugin selection form OK button now enables after Select All or other popup menu actions
- Refactored old log groups from MERGE to PATCH
- Added Bash Tag settings for Skyrim for every Bash tag that exists for Oblivion.
- No longer creating subfolders for smashed patches when the patch destination directory is the data folder
- Changing the smash setting now marks the patch as needing to be rebuilt
- Added groups to the Smash Settings manager.  If a setting's name has a dot in it, the text before the dot will be used as the setting group.  Groups are collapsible, and are reflected in filenames.
- Added special tag group parsing, if a tag starts with some text and then a colon, that text plus a period is prepended to each sub-tag.  E.g. {{Bash:Relev,Delev}} searches for the settings Bash.Relev and Bash.Delev.
- Added more hint information to the Settings Manager Tree view node hints for the Treat as Single and Preserve Deletions flags.

 

v0.1.1

- Added Select Similar Nodes
- Autoset attributes now disables IDLE records because smashing them commonly leads to failure
- Now updates count elements automatically, so no more CTDs from nasty NPC_ records with invalid COCT - Count elements.
- Fixed sorted array index issue for destination element when there were additions at earlier indexes
- Fixed VMAD treat as single not working properly
- Added smash setting hashes
- Added tag recognition for normal tags, extended tags, special extended tags, and multiple tags
- Fixed the combine setting option
- Fixed ITM removal removing non-ITMs
- Smash settings are now stored in game-specific directories.  You'll want to move your smash settings to a directory for your game.  So `.\settings\\`, game being Skyrim, Fallout3, FalloutNV, or Oblivion.
- Fixed a bug that caused the program to freeze while saving plugin errors
- Now conflict resolving with the winning master record
- Fixed loaded ESPs excluded from patch creation still influencing patch contents
- Improved the Edit Patch form to be more like the Edit Merge form from Merge Plugins
- Now saving logs in subfolders as it is done in Merge Plugins
- Changed DetailsListEditor to a virtual StringGrid, as it is in Merge Plugins
- Now offers an option to merge conflicting settings from the msConflictForm
- Fixed issue with autoset attributes not removing flags when disabling a child

 

v0.0.7

- Now copies winning override of overrides in files being smashed, which fixes a bug which caused partial load order smashed patches to often fail
- Fixed record prototype construction, now assigns to empty arrays so the record tree should always be complete now
- Build tree from plugins now targets records instead of top level groups.  You can now access and smash any record type.
- There are no more hardcoded record type exclusions.  You can now smash whatever you want.  Officially distributed settings will skip records that shouldn't be smashed, of course.
- Fixed exception when right clicking not on a node in tvRecords
- Fixed flag drawing bug that would draw flags in the upper left hand corner of tvRecords when expanding a node
- Splash screen now has glow on the loading text so it is always readable
- Now doing a better job of skipping records that don't need to be conflict resolved
- Progress bar is now more accurate when smashing patches.
- Now removing ITM groups when removing ITPOs
- Created Autoset Attributes option
- Created an "Automatic" smash setting which is built using Autoset attributes on a setting built from a full load order of plugins.

 

v0.0.6

- Preserve deletions in top-level elements (subrecords) now works

- Some improvements to logging

- Linking elements fixed

- Single entity element copying fixed

- Array identification conditions fixed

- Did away with the old building trees methodology. You can now right-click on the records node of a setting and use the Add submenu to create definitions for records.

- Added a new build tree option, which can be used from the right click popup menu.

 

 

v0.0.5

- Added code so you can now toggle multiple nodes in the tree using the space bar.

- Fixed Treat As Single Entity never being triggered because children nodes were always disabled and couldn't be toggle without removing the flag.

- Fixed chain nodes linking nodes at different levels, which also led to access violations when trying to remove the links.

- Failed smashed patch status now retains after smashing completes.

- Smashing is now cancelable.

- Now storing element definition type information in setting trees.  NOTE: Elements in old setting trees will have the "Unknown" type.  You'll need to rebuild your settings to get type information.

- Made prune nodes option more user friendly and added an auto-prune option to the popup menu for fast and simple pruning.
- Fixed bug with deleting a smash setting not removing it from disk.  They now get sent to the recycle bin when deleted.

- Fixed issue with processing records when they were set to not be processed in the setting (which created ITPOs).

- Distinguishes between arrays better.  Unsorted dtArray elements will now be treated as arrays, as intended.

- Removes ITPOs after smashing.

 

v0.0.4

- Added -profile param handling
- Algorithm now supports multi-linking (chaining) elements
- Added option to chain nodes in the Settings Manager
- Hardcoded excluded groups are now only RACE, NAVI, and NAVM
- Cloning settings will no longer create an infinite loop
- Plugin selection form now has a popup menu with Select all, Select none, and Invert selection options
- Plugin selection form now has extra options for targetting/skipping records
- Fixed issue with building tree not assigned TElementData for the rootNode, causing an access violation when checking anything in the tree

 

v0.0.3.14

- Core functionality is complete

- The UX for smashing and the algorithm are both more-or-less complete, may be updated

- Most of what needs to be done now is the creation of smash settings, which can be done by anyone with some knowledge of patching

- I'll try to make some documentation/tutorial resources for creating smash settings so you guys can get started on them

 

 

 

 

Purpose
Mator Smash is an application built on the xEdit framework developed to replace the primary functionality of Wyre Bash - the creation of patch files to combine (or merge) conflicting overrides to overcome Bethesda's "Rule of One". Wyre Bash used to provide this functionality in the days of Oblivion through the usage of bash tags, keywords that would be put in the description of the mod that would then tell Wyre Bash what types of records it should try patching. However, the current status is that Wyre Bash only provides this for a handful of record types for Skyrim, and this has been the case for quite a while now. There are several users (most notably Sharlikran) who have been working to update Wyre Bash's codebase to work with Skyrim, but it's been an uphill battle. As of now, it's not certain whether Wyre Bash will ever offer the same functionality it did in the days of Oblivion for Skyrim mods. That's where Mator Smash comes in.

 

Why is this necessary?
As per Skyrim's "Rule of One", any two mods that modify the same record but in different ways will conflict with each other in a way that can't be resolved asides from the creation of a compatibility patch. Because the number of Skyrim mods exceeds 40,000, there is no conceivable way to manually build enough compatibility patches to make every possible load order (or even a small fraction of them) work with each mod operating as intended. Mator Smash automates the most important part of the task of compatibility patch creation - the combining of conflicting edits. This is something that we've been needing for a very, very long time.

 

How does it work?
Warning: technical jargon ahead. Where Wyre Bash has separate procedures for the patching of individual record types, Mator Smash has a few generic procedures for the patching of ALL record types. This means that Mator Smash has, with very little development effort, achieved the capacity to patch more records than Wyre Bash has ever been capable of patching. This is achieved through a recursive traversal method which traverses override record structure, comparing subrecords between a master record, a source override record, and a destination override record. Upon the basis of this comparison, certain subrecords are written to the destination record while others are skipped and others are deleted from it.

 

Development Status
I began development of Mator Smash as a standalone application built on top of the xEdit API 10/18/2015. Here's the GitHub repository. Check out the design folder in the repository for various files relating to my design process. I'll be live streaming most of my development on livecoding.tv. I'll be streaming mainly on weekends 9:00AM-9:00PM PST. Because of major code reuse from merge plugins standalone, smash will be entering public beta very soon.

Mator Smash is currently available as a proof of concept script (which is slow and lacking in features/a clean user experience). You can download v0.9.4 of that script here. This script can currently correctly patch a massive number of different types of records. FAR MORE than Wyre Bash or any other currently available solution, and more than any solution that has ever existed for Skyrim or any other game.

 

Implemented Features

• Dynamic, flexible settings: Smash settings are comprised of a tree which has information on how records/subrecords should be handled. This will allow users to control how a patch is created on a per-subrecord basis. This will make smash relevant regardless of the nature of the mods being patched, or the game the mods come from. Each node (record/subrecord) will have the following data associated with it-
  • Process: Whether or not smash should process this node and perform conflict resolution for it if a conflict is found.
  • Preserve Deletions: If a mod deletes an element inside of this node, that deletion will be preserved in the generated patch.
  • Treat as single entity: If a mod overrides any subrecord in this node, the entire node will be replaced with that mod's version of it.
  • Chaining: Linking allows you to copy multiple side-by-side elements when any one of those elements changes.
  • Override Deletions: Allows a plugin to restore deleted elements.
  • Force Value: Forces values from a particular plugin to be used, and only allows plugins which require this plugin to perform further conflict resolution on affected records.
• Tags: As of v0.2.1, all Bash tags that were offered with Wrye Bash for Oblivion are now available for Skyrim.  Smash will automatically detect tags and apply smash settings to the plugins that have them, assuming you have smash settings that correspond to the tags.
• Managed smashed patches: Patches you build will be tracked in the program, so you can quickly and easily rebuild them when your load order changes. You can also have multiple patches for your load order, if you feel that is desirable.
• Multi-language support: I built a really clean, extensible, and easy to use language system when I developed Merge Plugins Standalone. Smash will use the same language system to support multiple languages out-of-the-box. As a translator, all you have to do is make a text file to create a translation and have users install it in their lang folder. The rest is handled by the program.
• Fast and easy setting creation: Select some plugins to build a setting to specifically handle the conflicts they are capable of creating in a load order.  Toggle or set flags for multiple nodes at once.  Select similar nodes in a setting tree.  Conflict resolution has never been quite so powerful, flexible, and easy.

 

Planned Features

Smash is now pretty much feature complete.  Further development energy will be spent on porting it to a zEdit application mode, zSmash, which will include a new, streamlined workflow.

[Mator]

@Mator I'm a little confused. Isn't the functionality of Smash essentially the same as that of Merge Plugins with the exception that that tool also merges the assets used? If that is the case what is the benefit you see of having two distinct tools when perhaps just one could suffice? Keep in mind my opening sentence.

No, absolutely not.

 

Merge Plugins does not do conflict resolution, it combines records in plugin files following the rule of one.

Mator Smash generates a compatibility patch which selectively combines edits on conflicting records based on user settings.

 

They are entirely different programs, and should be used together, not separately.  E.g. make a smashed patch for some plugins you plan on merging, and include it in the merge.  Make a smashed patch over your entire load order, including any merges plugins.

 

The way I view the difference between xEdit Merge and Wrye Bash/Flash is that the former merges based on some straightforward general purpose rules, while the latter allows user-created tags on individual plugin record types/categories to selectively alter the choices made at a record level when the merge is done. Isn't this the same as the difference between Mator Merge and Mator Smash? I also thought there was a lot of common code between the two Mator tools, but having common code doesn't always mean that two programs should be integrated. Perhaps these could be integrated, but the json files for each tag seem to have a fair amount of code specific to each tag type, so the programs might be better remaining separate.

 

Or perhaps I'm also confused.

It's important not to get confused between the different options available.

 

xEdit Merged Patch: Generates a compatibility patch for specific plugins, processing all conflicts in a fixed set of records/subrecords.  Always generates the same patch given the same records.  Generally unreliable, unable to handle a large number of conflicts.

 

Wrye Bash - Bashed Patch: Generates a compatibility patch for all plugins in the user's load order based on Bash Tags present in plugin descriptions.  The bash tags on plugins can be changed, but what the tags do cannot be changed (and new tags cannot be added without modifying the source code and recompiling the application).  Incomplete for Skyrim for many years now, support for some new tags was added just recently.

 

Mator Smash - Smashed Patch: Generates a compatibility patch for specific plugins, processing conflicts based on user specified "Smash Settings".  "Smash Settings" can be applied to plugins, combined, and can be automatically applied to plugins based on the existence of tags.  New smash settings can be created by the user, and existing smash settings can be freely modified.  Resolves whatever conflicts you want on whatever plugins you want - there's no limitation to what conflicts it can resolve or how it resolves them.

 

Merge Plugins: Combines the records in multiple plugin files following the rule of one.  That means the winning override record is what ends up in the merged plugin, with no exceptions.  This effectively creates a single plugin that operates identical to how the separate plugins operated in your load order without any other plugins present.  If you don't include compatibility patches in this merge, there will be subrecords that are lost and will not be retrievable by generating a compatibility patch using any tool (the conflicting data is gone - it no longer exists).  This is problematic for records that are overridden in multiple plugins that the game actually merges at runtime, like NAVI records.  Merging can also handle file-specific asset files such as FaceGenData, Voice Data, MCM Translations, SEQ Files, BSA Files, and Papyrus Script Fragments.

Edited February 4, 2016 by Mator

[Kelmych]

I don't remember the Merged Plugin doing any kind of sophisticated editing; I thought it picked the most recent entry for each object in the set of objects across the included plugins much like the game engine.

 

Edit: ninjaed

[Van555]

OK I have what may be a dumb question.

 

When you complete a Smash do you still need the plugins that it's made out of in your load order? I have to imagine so, but maybe you don't need the patch plugins anymore?

 

With wyre bash you can remove certain mods when you have the bashed patch completed. Is this at all similar?

 

[Mator]

I don't remember the Merged Plugin doing any kind of sophisticated editing; I thought it picked the most recent entry for each object in the set of objects across the included plugins much like the game engine.

That's what I just said.

 

When you complete a Smash do you still need the plugins that it's made out of in your load order? I have to imagine so, but maybe you don't need the patch plugins anymore?

Yes. 

 

With wyre bash you can remove certain mods when you have the bashed patch completed. Is this at all similar?

As of v0.2.2 there's a new feature which allows Smash to do the same thing. Read the changelog.

Edited February 4, 2016 by Mator

[GrantSP]

Thanks @Mator. I guess the similarities in the UI prompted the question more than the functionality. Now that it is stated clearly what each tool does, and doesn't do, there should be no further confusion.

[Mator]

Thanks @Mator. I guess the similarities in the UI prompted the question more than the functionality. Now that it is stated clearly what each tool does, and doesn't do, there should be no further confusion.

The UI is similar because it's easy to reuse code.  Why reinvent the wheel?  :P

 

The internal code also has a lot of similarities.  I'd say there's about 50% crossover between the applications, maybe more.

[Van555]

That's what I just said.

 

Yes. 

 

As of v0.2.2 there's a new feature which allows Smash to do the same thing. Read the changelog.

Thank you for your quick reply

[GrantSP]

If for any reason you wish to cancel Smash from executing it presents an error dialogue thus:

Application Error

Exception EOSError in module MatorSmash.exe at 000273E1
System Error. Code 5.
Access is denied.

Merge Plugins is able to exit calmly if 'Cancel' is pressed.

[EssArrBee]

I am unable to add CREA - Creature records to new settings with FNV. Error popup says 'invalid pointer operation'. Apparently they don't exist for Skyrim. Pretty sure Oblivion, FO3, and FNV have them though.  

[Mator]

Added both of those to GitHub issues and will have them handled by the next build.

[Kelmych]

I'm starting to test Mator Smash with Fallout 3; as with the other Bethesda games it certainly seems to have the potential to greatly simplify patching. Following an example in the Video Tutorial I made a New Setting and then selected Build and Build from plugins. When I do this I get an invalid pointer operation. There are 15 record types underneath Records added after hitting OK when receiving this error.

[EssArrBee]

We might have to go through the record types and see which ones give off that error. I know CREA records do already. I'd bet stuff that's specific to Fallout and not in Skyrim will have that error. 

[Mator]

To my understanding it's just the CREA records that do it.  As for why it does... that's a more complex question.  In order to figure out I'm going to have to step through the code.  It must be some interaction between the Fallout 3 definitions and my code which is different from any other record definition.

[Kelmych]

I created a setting for ALCH objects in Fallout 3 for Mator Smash using the default values. It works quite well at handling the Effects records in the objects included in the patch. I have a few questions:

• When creating the settings for ALCH Mator Smash shows EFID and EFIT subrecords under Effects, but not Conditions and CTDA subrecords. Is this because the Effects processing doesn't need to include different behavior for CTDA subrecords since they are adequately handled by the overall Effects processing?
• the Mator Smash patch included patches for only a small set of ingestible objects (5 with the mods I used from the Clear and Present Danger Fallout 3 guide). There are a lot of other ingestible objects that I expected to be included in the Mator Smash patch but are not. Some of the other ingestible objects conflicts seem very similar to those with the objects that were included in the patch. For example looking with FO3Edit, Purified Water has no condition subrecords in Fallout3.esm, several mods add condition subrecords (they happen to be the same condition subrecords), and the last plugin loaded (from the DIFF mod) has no condition subrecords. I expected the Mator Smash patch to include this object with the condition subrecords.

Thanks again for all the work you're doing on this.

 

If you're wondering why I am testing with Fallout 3 it is in part because it handles the same number off plugins as Skyrim (unlike Fallout NV), and like Fallout NV the Fallout 3 bashed patch handles a large set of bash tags (see the Advanced Readme) so I can compare Mator Smash and Wrye Bash/Flash results.

[Kelmych]

One of the common mod conflicts is caused by small plugins that change only a few records. Because of the "Rule of One" these plugins remove records/subrecords added by previous loaded plugins if they are loaded late in the load order, reminiscent of the "Preserving Deletions" option in Mator Smash object settings.  If they are loaded fairly early in the load order then their changes are ignored, unless there is a bash tag that specifically covers the record types they are changing.

 

Perhaps a new kind of mod tag could be added that applies to the whole mod. I was specifically thinking of a tag that causes Mator Smash to ignore all records/subrecords in the mod that are blank or are identical to the records/subrecords in the main game esm. Small plugins with this tag could then be loaded after the larger plugins that may change many records. The changes from the small plugin to a few records could be used, and they wouldn't disturb the records from other previously loaded plugins. It might eliminate or at least reduce the need for tags that otherwise need to be added to the current list of bash tags (e.g., for the Icon record in the Fallout series).