Kezyma's Root Builder for Mod Organizer

[Kezyma]

11 minutes ago, Mousetick said:

Yep. Redirect option. Got it.

I'm giving Root Builder a try. With 'Autobuild' switched off because I don't really need to swap root mods and I like being able to run several apps off MO2 simultaneously (GUI unlocked).

Is there a way to disable all the MO2 warnings for mods that only contain a root folder? Telling MO to ignore the warning during installation is fine, but then they show up as greyed out in the mod list with a X icon. It's only cosmetic, but it's distracting and annoying. It doesn't help make root mods feel like full citizens "at home" in MO2.

Which MO plugin is validating the contents, is that <Game Name> Plugin? Is there a way for your plugin to override this behavior?

Autobuild is relatively new, with it switched off, just make sure you remember to clear when you're done. It's probably fine, but it can be a bit annoying to clean up if your game happens to update while you've got a build in place  

That'd be nice, but there's no way for me to do that. The API available to plugins doesn't let me change the game plugin functionality. I'm not sure what exactly determines it though! I'm hoping that it gets expanded to allow this in future though! There's a lot of feature requests I've made to try and improve my plugins and I've used workarounds wherever possible in the interim. I will update it as new stuff gets added to MO though

[Mousetick]

46 minutes ago, Kezyma said:

I'm not sure what exactly determines it though! I'm hoping that it gets expanded to allow this in future though!

Apparently it's hardcoded in <Game Name> Plugin. Here for example the list of valid folders for Skyrim SE:

https://github.com/ModOrganizer2/modorganizer-game_skyrimSE/blob/master/src/skyrimsemoddatachecker.h

It might be a simple matter of adding "root" to that list. Unfortunately it's all C++ code so one can't just quickly hack it without rebuilding everything from source code.

But as you can see the MO2 developers already have included folders in that list like 'skse', 'MCM' and 'NetScriptFramework' which are not standard Gamebryo folders, so hopefully it shouldn't be a problem for them to add 'root' if you ask nicely 

[Kezyma]

9 hours ago, Mousetick said:

Apparently it's hardcoded in <Game Name> Plugin. Here for example the list of valid folders for Skyrim SE:

https://github.com/ModOrganizer2/modorganizer-game_skyrimSE/blob/master/src/skyrimsemoddatachecker.h

It might be a simple matter of adding "root" to that list. Unfortunately it's all C++ code so one can't just quickly hack it without rebuilding everything from source code.

But as you can see the MO2 developers already have included folders in that list like 'skse', 'MCM' and 'NetScriptFramework' which are not standard Gamebryo folders, so hopefully it shouldn't be a problem for them to add 'root' if you ask nicely 

I have asked about it before. It'd need adding to more than just the skyrim plugin, since it works for all Bethesda games! It'd also probably need a content indication icon. I can't quite remember why it wasn't done, but I've asked again. It'll be a while either way! I could probably do it myself and just open a PR, but I don't want to go messing with other peoples code! Ideally they'd add a way for plugins to alter the current game's list of valid files/folders, but that's probably a long way off!

There's a lot of cool features I'd like to add, which is why RB is technically still in beta, but I'm taking a little break at the moment as I've written a lot of plugins in athe last month (Reinstaller 30th Nov, Shortcutter 10th Dec, Plugin Finder 15th Dec, Curation Club 22nd Dec, Profile Sync 26th Dec) and released a bugfix for the CC Bloodchill Manor with majestic mountains (and other landscape changing mods). Need to take some time to actually play the games haha!

[DoubleYou]

@Kezyma

All you need to do is be able to change the corresponding meta.ini file for the mod to have validated=true instead of validated=false. If you right-click a mod that is marked as missing data, if you select the "Ignore Missing Data" option, that's what it toggles.

[Kezyma]

10 minutes ago, DoubleYou said:

@Kezyma

All you need to do is be able to change the corresponding meta.ini file for the mod to have validated=true instead of validated=false. If you right-click a mod that is marked as missing data, if you select the "Ignore Missing Data" option, that's what it toggles.

Whatever checks that will re-run when you open and close the info panel in MO as far as I'm aware. At some point I'm sure the root folder will get added to the supported MO games! 

[Mousetick]

1 hour ago, Kezyma said:

I have asked about it before. It'd need adding to more than just the skyrim plugin, since it works for all Bethesda games! It'd also probably need a content indication icon. I can't quite remember why it wasn't done, but I've asked again. It'll be a while either way! I could probably do it myself and just open a PR, but I don't want to go messing with other peoples code! Ideally they'd add a way for plugins to alter the current game's list of valid files/folders, but that's probably a long way off!

Yes it would need to be added to all BGS games, but it doesn't seem like copy-pasting the "root" string into a dozen or so .h files would be an insurmountable task for the MO2 dev team, or for yourself via a PR. Making an API with Python bindings available to plugins is another story, and I can understand that they would be more hesitant to implement such a feature, not to mention it could be quite a while before it's ready.

In another topic here, you alluded to the 'root' folder being the "de-facto" standard for packaging root mods. That may be true, but the reality is that it's currently not supported by any mod manager, as far as I know - with MO, one of the most popular ones, among them. So there is no incentive for mod authors to package their root mods in a 'root' folder. If you want it to really become a de-facto standard, you need to start somewhere, and adding 'root' folder support to MO would be a good place to start a virtuous cycle.

There is a difference between 'must work as expected' and 'nice to have'. I'd put the support for 'root' folder for content validation in the first category, and the content indication icon in the second. Get the essential basic support in first, then it can be improved later.

Anyway, there's no hurry and I don't mean to pressure you in any way. The 'root' mods being considered invalid by MO2 is not a deal-breaker for me, and there are workarounds. But I'd be hesitant to recommend Root Builder to average users who typically have a low tolerance for unexpected behavior and/or are not willing to make any effort to deal with it.

[Kezyma]

10 minutes ago, Mousetick said:

Yes it would need to be added to all BGS games, but it doesn't seem like copy-pasting the "root" string into a dozen or so .h files would be an insurmountable task for the MO2 dev team, or for yourself via a PR. Making an API with Python bindings available to plugins is another story, and I can understand that they would be more hesitant to implement such a feature, not to mention it could be quite a while before it's ready.

Perhaps not, it was a long time ago that it was last discussed and I can't remember the exact reason it wasn't done. Likely because at the time, RB wasn't even an MO plugin. There's lots of stuff that the API needs extending to do, but the issue usually isn't that they're unwilling to do it, it's usually more of a lack of time and dev resources to get around to it. 

Quote

In another topic here, you alluded to the 'root' folder being the "de-facto" standard for packaging root mods. That may be true, but the reality is that it's currently not supported by any mod manager, as far as I know - with MO, one of the most popular ones, among them. So there is no incentive for mod authors to package their root mods in a 'root' folder. If you want it to really become a de-facto standard, you need to start somewhere, and adding 'root' folder support to MO would be a good place to start a virtuous cycle.

The Root folder is the standard as far as MO and MO plugins are concerned. It's not the standard outside MO though. By standard in this sense I mean that if you have your mods installed in a way that works with RB and then RB stops working, any future plugins for MO (or direct updates to MO itself) will use the same mod structure, as does the current alternative to RB! I wouldn't ever expect mod authors to pre-pack mods with a Root folder though, since the global 'default' for modding isn't to use a mod manager at all, let alone MO specifically!

All known MO plugins have been developed so closely with or by MO devs in the first place that the lines between official MO stuff and third-party plugins is pretty blurry. A lot are included with MO itself, but have newer versions available (we're currently planning on including MO 'official' plugins in Plugin Finder so that people can update them), and a good portion of them seem to be developed as third-party plugins and then get included in MO releases later on. Both Root Builder and Plugin Finder could end up in MO in future, but the MO plugin API needs extending so that I can replace some of the workarounds. They're also keen on RB defaulting to USVFS + Link mode instead of Copy mode, so that needs more testing and work first too. 

Quote

There is a difference between 'must work as expected' and 'nice to have'. I'd put the support for 'root' folder for content validation in the first category, and the content indication icon in the second. Get the essential basic support in first, then it can be improved later.

Anyway, there's no hurry and I don't mean to pressure you in any way. The 'root' mods being considered invalid by MO2 is not a deal-breaker for me, and there are workarounds. But I'd be hesitant to recommend Root Builder to average users who typically have a low tolerance for unexpected behavior and/or are not willing to make any effort to deal with it.

Currently both of those are considered nice to have but non-essential, since everything still works perfectly fine and warnings don't prevent any functionality. It's somewhat annoying to people who pay attention to it, but so far I've only had one user ask about it, and they were quite content when I told them they could ignore it (and updated the docs to let people know they can). I don't think most users even notice the warning in the first place. I also think the thing is that the users that aren't willing to put the effort in are also users that aren't bothering with MO in the first place, they're either installing manually or using Vortex because the Nexus tells them to. 

I've stuck a query in the MO discord about adding it to Bethesda games, so will see what is said about it! 

14 hours ago, Mousetick said:

I'm giving Root Builder a try. With 'Autobuild' switched off because I don't really need to swap root mods and I like being able to run several apps off MO2 simultaneously (GUI unlocked).

I've just checked and RB's autobuild doesn't clear until one of the apps is actually closed, so you can unlock the UI without clearing as long as you leave all the apps open. Just as a side note!

[Mousetick]

It doesn't sound like we're talking about the same things. I'd like to clarify some points.

1 hour ago, Kezyma said:

I wouldn't ever expect mod authors to pre-pack mods with a Root folder though, since the global 'default' for modding isn't to use a mod manager at all, let alone MO specifically!

Why not? If mod managers start supporting a root folder structure, mod users may start requesting that mod authors package their root mods as such, because it's easier to install and manage them. The mod authors may or may not listen, I don't know. Other mod managers may or may not follow suit, I don't know. But at least there is a path forward, some foundation to build upon. In the current situation, nothing will change and it remains as "difficult" to install a root mod manually without a mod manager, than with one.

What about FOMODs? They're packaged specifically for use with mod managers, which is not the global 'default'. They can still be installed manually if one wants.

1 hour ago, Kezyma said:

Both Root Builder and Plugin Finder could end up in MO in future, but the MO plugin API needs extending so that I can replace some of the workarounds.

It seems like you're shooting for the stars which is great. I'm only suggesting baby steps. All I'm suggesting is that a root mod with only a 'root' folder be considered a valid mod by MO, so that it can be installed and displayed in the mod list like any other mod without throwing warnings. That's all. Everything else remains the same for RB and MO.

Any improvement such as shipping RB as part of the MO package, or tighter and cleaner integration between RB and MO via enhanced APIs would be wonderful. But that can come later.

1 hour ago, Kezyma said:

I don't think most users even notice the warning in the first place.

The issue is that RB requires users to repackage the root mods so that the root files are in a root folder. This is not very complicated, but it's error-prone and worrisome if you're not a power user. And then the mod manager throws warning and tells the user this mod is invalid, which must be ignored to proceed. Once the mod is installed, the mod manager keeps telling the user the mod contents are invalid. How can the user tell that this is a false positive? Perhaps they did make a mistake in re-arranging the file/folder structure, but how would they know?

It's a bad user experience. Mod managers are supposed to make installing and managing easier and safer for users.

[Mousetick]

2 hours ago, Kezyma said:

I've just checked and RB's autobuild doesn't clear until one of the apps is actually closed, so you can unlock the UI without clearing as long as you leave all the apps open. Just as a side note!

Let's say I have autobuild turned ON and I launch app1. Later I launch app2. Later I quit app1. RB will perform a clear which may affect app2 still running. Later I enable or disable a root mod. Later I launch app3. RB will perform a build which may affect app2 still running. I'd rather keep autobuild OFF and not have to worry about what may happen. It's just my personal preference, so I'm glad the Autobuild can be switched off and thanks for making it possible.

[Kezyma]

16 minutes ago, Mousetick said:

It doesn't sound like we're talking about the same things. I'd like to clarify some points.

Why not? If mod managers start supporting a root folder structure, mod users may start requesting that mod authors package their root mods as such, because it's easier to install and manage them. The mod authors may or may not listen, I don't know. Other mod managers may or may not follow suit, I don't know. But at least there is a path forward, some foundation to build upon. In the current situation, nothing will change and it remains as "difficult" to install a root mod manually without a mod manager, than with one.

I mean they 'could' end up doing that, I just don't 'expect' it. It's mostly out of my hands! Ideally, if I can ever figure out a way to identify the appropriate install structure of root mods, I'd just hook into MO's install process and repackage them so it ceases to be an issue for install complexity, but that's a long way off!

Quote

What about FOMODs? They're packaged specifically for use with mod managers, which is not the global 'default'. They can still be installed manually if one wants.

As you said, FOMODs can still be installed manually, and I'd probably guess that they're installed manually quite often! The only real exception is stuff like OMODs, which require OBMM to install. Even then though, most people just use OBMM to install the OMODs and nothing else.

Quote

It seems like you're shooting for the stars which is great. I'm only suggesting baby steps. All I'm suggesting is that a root mod with only a 'root' folder be considered a valid mod by MO, so that it can be installed and displayed in the mod list like any other mod without throwing warnings. That's all. Everything else remains the same for RB and MO.

Any improvement such as shipping RB as part of the MO package, or tighter and cleaner integration between RB and MO via enhanced APIs would be wonderful. But that can come later.

Yeah I understand, it's just that's a decision out of my control. The answer I got when I asked about it today was that ideally they want plugins to be able to register additional folders as valid instead of adding it directly to the plugin. We'll see what happens with that next! Either way though, it wont be for another MO version or two before it gets implemented!

Quote

The issue is that RB requires users to repackage the root mods so that the root files are in a root folder. This is not very complicated, but it's error-prone and worrisome if you're not a power user. And then the mod manager throws warning and tells the user this mod is invalid, which must be ignored to proceed. Once the mod is installed, the mod manager keeps telling the user the mod contents are invalid. How can the user tell that this is a false positive? Perhaps they did make a mistake in re-arranging the file/folder structure, but how would they know?

It's a bad user experience. Mod managers are supposed to make installing and managing easier and safer for users.

Yeah, I'd like to find ways to make both easier. An automatic install process to restructure mods would be ideal, but complex to write and possibly error prone. In terms of MO warnings, it's just something that has to be worked around until MO gives me a way to resolve it. For now, I just make a note of it in the docs so that any users reading them for answers will know it's probably okay! They know I want a way to stop those errors, but until someone actually sits down and implements it, I'm sadly a bit stuck.

RB has been developed for years now, and in terms of usability, it's drastically improved. The early versions required installing RB separately, adding it as an app to MO, repacking all your mods in root folders, launching RB through MO, running a (very slow) build to copy stuff across, then doing similar to clear later. There was little to no handling of file changes or additions and game updates would totally break it. It wasn't until about 6 months ago that it finally got rewritten as a plugin and got features like autobuild and redirect to make it user friendly. Now, the only thing a user has to do is learn how to repack a handful of their mods, and assuming they refer to the docs if/when they get stuck, they mostly seem okay!

If I can make it easier to use, without sacrificing functionality or introducing fragile workarounds, I will, I'm just waiting on MO to get updates that allow for it.

[Kezyma]

21 minutes ago, Mousetick said:

Let's say I have autobuild turned ON and I launch app1. Later I launch app2. Later I quit app1. RB will perform a clear which may affect app2 still running. Later I enable or disable a root mod. Later I launch app3. RB will perform a build which may affect app2 still running. I'd rather keep autobuild OFF and not have to worry about what may happen. It's just my personal preference, so I'm glad the Autobuild can be switched off and thanks for making it possible.

Building a second time while the first is still in place will just add any new files to the build, but wont remove anything you've disabled in the meantime. It'll clear when you close an app and build when you launch one. It could get confusing though and I understand, I was just pointing out that you can technically run multiple apps while having MO unlocked without clearing as I only just realised it was possible. I believe there's already some issues that can arise from running multiple apps through MO and changing enabled/disabled states anyway, which aren't to do with RB. I think that's why MO locks itself in the first place and encourages only running one app at a time.

Autobuild is new, most users coming from older versions are used to manually building and tend to turn it off to stick with the workflow they're familiar with! They're all just tools and I'm glad you've found a way to make it work for you!  

[z929669]

Thinking aloud here and sharing:

I just want to point one thing out with respect to this discussion about a 'root' folder. For Bethesda games, mod managers by default understand 'root' as the directory in which (most) mods are installed ... so ../Data ... 'root' mods are one level above this, so mod managers don't handle the installation properly.

To me, discussions of a root folder lead us (me?) to think about this literally as ../root. I believe what is being discussed is a change in this default 'root' definition such that it is defined at the {game} folder level, so ../{game}/Data (not ../{game}/root/Data)

A mod packaged such that its 'root' definition is implicit as ../Data could be repackaged one level up. So the original zipped file may contain meshes/. and textures/. as top-level folders, and the repackaged top level would be Data/meshes/. and Data/textures/.

I believe MO and most other worthy mod managers gracefully recognize either structure and handle them accordingly? The folder structure always implicitly defines the root as ../Data and if that is present in the package structure, then this level is ignored?). Therefore (if so), I don't see why 'root' needs to be added to the list of valid folders in https://github.com/ModOrganizer2/modorganizer-game_skyrimSE/blob/master/src/skyrimsemoddatachecker.h Why not just NOT ignore ../Data instead? So a root mod would always have Data/ and files beside it. The Data folder could be empty or have valid files/folders under it (but always Data/ and always one or more files beside it).

I'm not that familiar with the code-base definitions/functions used by the various mod managers, but having done some very basic programmatic code alterations along these lines, I think it should be trivial to add an implicit "root check" and create a branch for the existing method and a new branch for the RB method in the mod-manager code? Non-Bethesda games would use a different folder-name basis, but I would think that they all have an analogous one like Data/

In this way, only root mods would need to be repackaged (or include some metadata definition), and all other mods would be implicitly conveyed down the 'old' code branch.

---

Anyway, I don't have time or inclination to get sidetracked down yet another rabbit hole in testing this, but these were my thoughts in reading through this discussion. Perhaps helpful or more likely just noise

[Kezyma]

50 minutes ago, z929669 said:

Thinking aloud here and sharing:

I just want to point one thing out with respect to this discussion about a 'root' folder. For Bethesda games, mod managers by default understand 'root' as the directory in which (most) mods are installed ... so ../Data ... 'root' mods are one level above this, so mod managers don't handle the installation properly.

To me, discussions of a root folder lead us (me?) to think about this literally as ../root. I believe what is being discussed is a change in this default 'root' definition such that it is defined at the {game} folder level, so ../{game}/Data (not ../{game}/root/Data)

A mod packaged such that its 'root' definition is implicit as ../Data could be repackaged one level up. So the original zipped file may contain meshes/. and textures/. as top-level folders, and the repackaged top level would be Data/meshes/. and Data/textures/.

I believe MO and most other worthy mod managers gracefully recognize either structure and handle them accordingly? The folder structure always implicitly defines the root as ../Data and if that is present in the package structure, then this level is ignored?). Therefore (if so), I don't see why 'root' needs to be added to the list of valid folders in https://github.com/ModOrganizer2/modorganizer-game_skyrimSE/blob/master/src/skyrimsemoddatachecker.h Why not just NOT ignore ../Data instead? So a root mod would always have Data/ and files beside it. The Data folder could be empty or have valid files/folders under it (but always Data/ and always one or more files beside it).

I'm not that familiar with the code-base definitions/functions used by the various mod managers, but having done some very basic programmatic code alterations along these lines, I think it should be trivial to add an implicit "root check" and create a branch for the existing method and a new branch for the RB method in the mod-manager code? Non-Bethesda games would use a different folder-name basis, but I would think that they all have an analogous one like Data/

In this way, only root mods would need to be repackaged (or include some metadata definition), and all other mods would be implicitly conveyed down the 'old' code branch.

---

Anyway, I don't have time or inclination to get sidetracked down yet another rabbit hole in testing this, but these were my thoughts in reading through this discussion. Perhaps helpful or more likely just noise

Mod Managers treat Game\Data as the 'base folder' that every mod goes into. Every mod in MO is packaged so that the contents of each mod map directly to Game\Data.

Root Builder lets you install mods directly to Game\, but because there top level for MO is not that high, I created a subfolder, which, if mapped directly onto the game itself via MO would be Game\Data\Root but then Root Builder maps that folder to Game\ (although obviously in a very different way due to the different requirements of those kinds of mods).

All mod managers I know will identify if a mod is packed with a Data folder or just as direct contents of a Data folder (based on a hardcoded list of valid folder names and file types). It then repackages all of them so that the mod only contains the contents of the Data folder and nothing higher than that. It's handled on install to MO as opposed to later on.

Root needs to be added to the list of valid data folders for three reasons.

• When installing a mod via MO, if someone hypothetically did package a mod with the Root folder structure, MO would automatically detect that and install it correctly without manual rearranging needed.
• When installing a mod via MO, the red border and warning message that your mod is packaged wrong during manual installation would go away once you had arranged it for Root Builder.
• When a mod is already installed, if it only contains a Root folder, MO would no longer show an icon next to it and grey it out with the message that it contains no valid files!

 

[Kezyma]

On 12/28/2021 at 3:52 PM, Mousetick said:

It doesn't sound like we're talking about the same things. I'd like to clarify some points.

Why not? If mod managers start supporting a root folder structure, mod users may start requesting that mod authors package their root mods as such, because it's easier to install and manage them. The mod authors may or may not listen, I don't know. Other mod managers may or may not follow suit, I don't know. But at least there is a path forward, some foundation to build upon. In the current situation, nothing will change and it remains as "difficult" to install a root mod manually without a mod manager, than with one.

What about FOMODs? They're packaged specifically for use with mod managers, which is not the global 'default'. They can still be installed manually if one wants.

It seems like you're shooting for the stars which is great. I'm only suggesting baby steps. All I'm suggesting is that a root mod with only a 'root' folder be considered a valid mod by MO, so that it can be installed and displayed in the mod list like any other mod without throwing warnings. That's all. Everything else remains the same for RB and MO.

Any improvement such as shipping RB as part of the MO package, or tighter and cleaner integration between RB and MO via enhanced APIs would be wonderful. But that can come later.

The issue is that RB requires users to repackage the root mods so that the root files are in a root folder. This is not very complicated, but it's error-prone and worrisome if you're not a power user. And then the mod manager throws warning and tells the user this mod is invalid, which must be ignored to proceed. Once the mod is installed, the mod manager keeps telling the user the mod contents are invalid. How can the user tell that this is a false positive? Perhaps they did make a mistake in re-arranging the file/folder structure, but how would they know?

It's a bad user experience. Mod managers are supposed to make installing and managing easier and safer for users.

I've just released v4.3.3 which includes an incredibly experimental and rather untested plugin which you can enable to attempt to automatically detect and install root mods. Probably wont work with everything, but I'll improve support as broken mods are reported to me!

[D1Z4STR]

Sorry to resurrect this thread but I am indeed interested in using Root Builder. I have 2 different mod lists installed for Skyrim AE using Wabbajack (WJ). I love WJ... But, I also would like to utilize STEP for Skyrim AE. Since STEP uses the main game directory for files, etc. that would cause my WJ mod lists to not work. So what choices do I have? The answer is none unless Root Builder is used (last updated last month). All the mod lists are using it these days and I wanted to ring the bell to see if STEP would think about it. Thank you.