ReLinker

[Ganda]

ReLinker

Hishy had the idea of making a little tool to allow us to merge mods with scripts - it takes the merge maps that are produced when you merge something, grabs all the scripts and LODGen files from you active mod list (BSAs included) and replaces the old plugin name and formIDs with the new plugin name and formIDs.

We all know what a fantastic tool Merge Plugins is. However there are certain mods which cannot be merged due to the use of certain scripting functions which are called on the original plugin name. Functions including GetFormFromFile, GetModByName, checkModPresence, and IsPluginLoaded look for specific .esp names. When you create a merge the original .esp name(s) no longer exists (unless you rename the merged plugin back to one of the original plugin names but this in itself can cause problems including broken MCM menus and is therefore not ideal) and therefore any function call looking for them will no longer trigger. These functions can be called by one of the scripts included in the mod being merged, or by scripts from other mods outside the merge.

A prime example of this is how the 'self-patching' in Legacy of the Dragonborn works by looking for the existence of a whole host of plugin names and activating the various additional displays if the mods are loaded. ReLinker actually came about out of a conversation regarding merging the four BadGremlins treasure hunt mods and patching the LotD activator script to make sure all four displays were still activated.

ReLinker is run outside ModOrganizer and runs on whichever profile you last had open before closing out of MO. It scans your load order looking for merges, and for every merge it finds it reads the contents of plugins and map .txt files and creates a reference list of the original plugin names and the new merged plugin name, along with the original and renumbered FormIDs. It then scans every script in your load order looking for any of those functions mentioned above and editing the scripts to reflect the new plugin names and formIDs. So in the example above once we've merged all four treasure hunts into Treasure Hunts.esp and run ReLinker, the DBM script will now look for Treasure Hunts.esp four times and activate all four displays. Likewise if you were to merge a mod like Sneak Tools whose scripts contain numerous Sneak Tools.esp lookups, ReLinker would then edit all of the Sneak Tools scripts in the merge itself to ensure they now look for the new merged plugin name and renumbered formIDs instead.

ReLinker works on both loose files (if you unpack your BSAs) and on scripts within BSAs. It directly edits the .pex files, doing away with the need to decompile and recompile the scripts which has greatly sped up the process over in the release version.

Instructions:

1. Make sure you have the profile you want to ReLink active in Mod Organizer and that you have completed and activated all of your merges.
2. Exit Mod Organizer
3. Run ReLinker and follow the on screen instructions to specify the required paths.
4. Check on screen output to ensure ReLinker has completed successfully.
5. Close the ReLinker window and open Mod Organizer.
6. Scroll to the bottom of the left pane and locate the new ReLinker Output mod.
7. Activate ReLinker Output and leave it at the bottom of the left pane to ensure the new ReLinked scripts have priority over the originals.
8. That's it, there is no 8

If you add/remove/update any mods or create any new merges and want to ReLink again then simply remove the existing ReLinker Output mod completely, exit MO and repeat the process again.

Prerequisites:

• Mod Organizer - does not work with other managers;
• Do not remove the merge folder after completing a merge.

Download it here. The tool will ask you for some things and will try to guess them properly. If the values between square brackets are correct, simply press ENTER without inputting any value. The output scripts and LODGen files will be stored in a new mod folder named "ReLinker Output". To revert the changes from this tool simply disable or delete the folder.
 
Good Practices:

• ReLinker will run with the latest profile used in MO;
• Close MO before running it.

By default, the following BSA's are ignored to speed up the process and because they don't contain any scripts with the functions that are affected:

• Skyrim - Misc.bsa
• Skyrim - Shaders.bsa
• Skyrim - Textures.bsa
• Skyrim - Interface.bsa
• Skyrim - Animations.bsa
• Skyrim - Meshes.bsa
• Skyrim - Sounds.bsa
• Skyrim - Voices.bsa
• Skyrim - VoicesExtra.bsa
• Update.bsa
• Dawnguard.bsa
• HearthFires.bsa
• Dragonborn.bsa
• HighResTexturePack01.bsa
• HighResTexturePack02.bsa
• HighResTexturePack03.bsa
• Unofficial Skyrim Legendary Edition Patch.bsa
• Falskaar.bsa
• Wyrmstooth.bsa

If you have specific issues or suggestions post it directly to the this topic. I'm always happy to receive questions and feedback. Happy merging!

 

EDIT: Thanks dunc for the much better description!

Edited February 20, 2017 by hishutup

[Ganda]

I'm not sure this entirely explains it.  Take the example of the Treasure Hunts merge for the five treasure hunt mods which integrate into Legacy of the Dragonborn.  For the displays to be enabled the LotD enabler script looks for the original esp names.  If all five are merged into one new esp then obviously when LotD looks it can't see them and the displays remain disabled.  What ReLinker does is not simply an extension of Merge Plugins.  It scans your load order for merges, then reads the plugins.txt and map.txt in each merge. Then it looks at ALL of the other scripts in your load order looking for any instance of any of the original esp names for mods now combined in any of the merges.  When it finds an instance it then changes it to point at the new merged plugin esp and edits any related formIDs based on the relevant map.txt.  This way LotD now knows that Treasure Hunts Merged.esp now contains all five original esps and the enabler script is updated accordingly allowing the displays all to enable.

 

TLDR - ReLinker edits any script in your load order (not just those which are part of a merge) which is looking for an esp now contained in a merged plugin to point at the merge instead.

Dunc, do you mind making a "tiny" explanation to add to the OP and to the nexus?

Edited August 16, 2016 by Ganda

[Darth_mathias]

nice to see this released guess I will have to finally get up off my ass and give it a try.

[hishutup]

Why I think MP should not handle this.

 

The billboards, yes but for the scripts, probably not. Its kind of hard to explain why but treat it like bash but instead of records, its scripts. Let's say you have an interesting merge where plugins a, b, c, d and e reference each other. If someone merges b and c together(BC), then merges d and e(DE). Also, let every formid be altered. If mp handled this in the merge, only a and BC will work, BC would know nothing of DE and vice-versa. However, treating it like a "autopatcher" avoids this issue.

 

As for smash, why not that's up to mator, Ganda is pretty open.

When you say "scripts" that MergePlugins doesn't handle, do you mean scripts that use the GetFromForm and/or GetFromFile functions (the functions that hardcode the ESP/ESM filename into the script)?

If so, that is awesome, because that is the last piece that Mator's MergePlugins standalone doesn't handle (since he added script fragment handling).

Yup, it handles those and a few custom functions. sometimes the compiler fails to compiler, I found a few condition because of skyUI which I should add to the op.

[Ganda]

Ganda is pretty open.

Phrasing!

 

Yup, it handles those and a few custom functions. sometimes the compiler fails to compiler, I found a few condition because of skyUI which I should add to the op.

Read the op again hishy. Also, comments are waiting for you in the nexus

[hishutup]

Phrasing!

Nah...

 

 

Read the op again hishy. Also, comments are waiting for you in the nexus

Nah...

No, comments are waiting for you.

[dunc001]

Dunc, do you mind making a "tiny" explanation to add to the OP and to the nexus?

Sure, np as long as you don't need it yesterday...

[Mator]

In a nutshell, this "finishes" the merge by merging what MergePlugins can't merge (the scripts and LODGen)

 

EDIT: As I said before, I made this on hishy's instructions. See this post.

Slightly inaccurate.  The only scripts Merge Plugins "can't merge" are ones with GetFormFromFile calls (also GetModByName, checkModPresence, and IsPluginLoaded calls).  That's what you're handling here.  Merge Plugins handles script fragments and ordinary scripts don't need any special handling.

 

EDIT: Seems other people covered that already.

 

I explained all of this to hishy in depth at some point.  It's cool that he managed to remember the details enough to get someone to make a solution.

 

Merge Plugins COULD handle it, but it would have to function similar to how Ganda has this programmed in that it'd have to check all scripts after each merge because each merge can require the alteration of any number of scripts from any number of other mods/merges.

 

ANYWHO...

 

Really awesome to see this!  I wanted to program it but I've been programming my ass off on Mod Picker so I haven't had the time.  Nice work Ganda, I'll check out the code sometime soon-ish.  :)

 

EDIT 2:

 

The patch is only needed for compiling when running the CK through MO. Else the compiler in the CK will fail. It's not likely needed in any other situations such as this.

 

I kinda see this being absorbed into Mator's tool since that is just local. Would you be okay if he added this into Mator Smash? Though I don't think he's worked on that tool in a long time. Not since he started his website project.

I think it'd be better to absorb it into Merge Plugins because Smash doesn't have the hooks into the Papyrus Compiler/Decompiler integrations yet.  It'd be a "post-merge fix things patch".  The LODGen stuff can be handled immediately during the merging process like other file specific assets though.

 

 

EDIT 3:

Line 278 LMAO.

 

This feels a bit inefficient.  The way I'd do this is to process all the script files once and build a dictionary of all the GetFormFromFile, GetModByName, checkModPresence, and IsPluginLoaded calls in all scripts, then iterate over that dictionary and for each dictionary item iterate across all the maps.  Or just combine all the maps into a single map and use a simple indexOf call on a single map to see if a plugin is merged.  The main reason this would be faster is you only have to load the script files once to build the dictionary of the calls, then load them once more to actually replace the calls as necessary.  Right now you're loading the files n times, where n is the number of maps in merge_data_list, which is a lot of Disk I/O which is slow (even with tiny files).

 

Alternatively you could also just reverse the loop order (so iterate over the files and per file iterate over the maps).  This would force the LODGEN to be a separate loop as well.

 

Looks pretty good overall though.  Only other thing I'd suggest is breaking things into more functions so you don't have quite as much nesting going on and so the main procedure is abstracted a bit.

Edited August 16, 2016 by Mator

[Ganda]

Line 278 LMAO.

Just for you

[Mator]

Are you working on the Nexus page still, Ganda?  It came up on google search results but said the author has hidden it.  :'(

 

Let me know when it's ready so I can pop in and give my thumbs up/endorsement. 

 

 

EDIT: Also, are you planning on posting this to r/skyrimmods?

Edited August 16, 2016 by Mator

[Ganda]

This feels a bit inefficient.  The way I'd do this is to process all the script files once and build a dictionary of all the GetFormFromFile, GetModByName, checkModPresence, and IsPluginLoaded calls in all scripts, then iterate over that dictionary and for each dictionary item iterate across all the maps.  Or just combine all the maps into a single map and use a simple indexOf call on a single map to see if a plugin is merged.  The main reason this would be faster is you only have to load the script files once to build the dictionary of the calls, then load them once more to actually replace the calls as necessary.  Right now you're loading the files n times, where n is the number of maps in merge_data_list, which is a lot of Disk I/O which is slow (even with tiny files).

 

Alternatively you could also just reverse the loop order (so iterate over the files and per file iterate over the maps).  This would force the LODGEN to be a separate loop as well.

 

Looks pretty good overall though.  Only other thing I'd suggest is breaking things into more functions so you don't have quite as much nesting going on and so the main procedure is abstracted a bit.

On my defense, I did that at 3 am before a 12 hour trip :p But yes, it could be optimized a lot more, the code is SUPER fugly and it's a bunch of spaghetti, I'll get to it after some other things I need to post.

 

The combine merge maps thing, python's dictionaries are unordered (and I don't like using ordereddicts on these cases) so merges of merges would get burnt :/ I'll find a way

 

Are you working on the Nexus page still, Ganda?  It came up on google search results but said the author has hidden it.  :'(

 

Let me know when it's ready so I can pop in and give my thumbs up/endorsement. 

 

 

EDIT: Also, are you planning on posting this to r/skyrimmods?

I somehow deleted the page o.0 Hishy will make a new one, since he's the brains on this one. and I never really posted anything on skyrimmods or ever been there tbh :/ All I've heard about was a couple of years ago that the community over there was worse than nexus, is it better now?

[Mator]

On my defense, I did that at 3 am before a 12 hour trip :p But yes, it could be optimized a lot more, the code is SUPER fugly and it's a bunch of spaghetti, I'll get to it after some other things I need to post.

For sure man.

 

The combine merge maps thing, python's dictionaries are unordered (and I don't like using ordereddicts on these cases) so merges of merges would get burnt :/ I'll find a way

I don't support merging merges in merge plugins. I feel like saying "if someone does that, **** them", but building your code for the lowest common denominator is always a good idea.

 

I somehow deleted the page o.0 Hishy will make a new one, since he's the brains on this one. and I never really posted anything on skyrimmods or ever been there tbh :/ All I've heard about was a couple of years ago that the community over there was worse than nexus, is it better now?

Looking forward to seeing it back up! Silly Ganda deletes his own mods. XD

 

r/skyrimmods is arguably the best part of the Skyrim Modding community right now, in my opinion. I love hanging out there and interacting with the community there. I've also heard it was bad at some point, though it's been nothing but awesome while I've been there.  With regulars like Chesko, Sheson, Enai, myself, and others you should fit right in. :)

Edited August 17, 2016 by Mator

[Ganda]

Sorry mator (and whoever else this applies to) that I haven't answered yet, because hishy made me learn how to edit pex files directly which I dreaded (BUT IT WAS SOOO FUUUUUUUUN) so I spent the last day just doing that. I'll see to applying it to the relinker and I'll answer everyone whenever I get back from the very deep hole I'm about to put myself into. And thanks to the random nexus user, I have no idea who it is since that page was deleted, for suggesting this and sorry for thinking you were crazy.

[hishutup]

Sorry mator (and whoever else this applies to) that I haven't answered yet, because hishy made me learn how to edit pex files directly which I dreaded (BUT IT WAS SOOO FUUUUUUUUN) so I spent the last day just doing that. I'll see to applying it to the relinker and I'll answer everyone whenever I get back from the very deep hole I'm about to put myself into. And thanks to the random nexus user, I have no idea who it is since that page was deleted, for suggesting this and sorry for thinking you were crazy.

It was the guy that made the restringer.

[Ganda]

Ok guys, I need heeelp! I'd really appreciate it if someone could get me some scripts, both compiled and source files, that contain the following functions:

• GetModByName but without Game. before it;
• GetFormFromFile but without Game. before it;
• IsPluginLoaded;
• checkModPresence.

I need to check how these are stored. Thanks! :D

 

r/skyrimmods is arguably the best part of the Skyrim Modding community right now, in my opinion. I love hanging out there and interacting with the community there. I've also heard it was bad at some point, though it's been nothing but awesome while I've been there.  With regulars like Chesko, Sheson, Enai, myself, and others you should fit right in. :)

 

I'll give it a shot once I finish this! :p

 

It was the guy that made the restringer.

Wooow...

[hishutup]

Ok guys, I need heeelp! I'd really appreciate it if someone could get me some scripts, both compiled and source files, that contain the following functions:

• GetModByName but without Game. before it;
• GetFormFromFile but without Game. before it;
• IsPluginLoaded;
• checkModPresence.

For the last two, I think iNeed was the mod that used it.