Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[z929669]

@Tech

I think Tannin's new proposal would make MO behave more like other mod managers. There is a definitive install order (loose asset prioritization) and a plugin prioritization order, which controls associated BSA assets. With regard to textures, I see what you mean, and we can decouple those from the plugin (or extract only the textures maybe). With regard to most other assets (scripts come to mind), it would be optimal to let LOOT manage what we might otherwise break using custom asset prioritization.

 

We should still be able to guide the user through a logistical "installation" order that makes intuitive sense (i.e., STEP Sections). As long as we still have BSA extraction and the plugin-decoupling functionality (which we may not even need .... tough to say ATM), I think we should be good. In short, as long as MO can proxy the behavior of Wrye Bash, we are good.

[TechAngel85]

@TechAngel85: Why does the step guide need to be installed in a certain order? Is there documentation on which conflicts step resolves in a certain order and why?The installation order is only relevant if there is a conflict between two mods, otherwise the order there simply doesn't matter.For the cases where two mods conflict and the order of the resolution is important (not down to preference), step should probably be very very careful proposing anything other than the plugin-order anyway. Think about it from the perspecive of the mod author: If he knows his mod contains scripts that conflict with scripts from other mods and thus may break the game HE will have to ensure users install his mod correctly.He could either provide an installation guide and hope people follow it. In this case step should follow his suggestion.Or he could package the scripts and other assets in a bsa so that loot does the asset ordering to ensure users get the mod installed correctly.The latter will work for people who don't extract BSAs and don't use MO, which should be the majority. It should never be necessary to unpack a BSA (or overwrite its load order) to get a mod working! In this case step should not suggest users unpack the bsa and order it contrary to plugin order, that would basically be saying "we know better than the author".This leaves mods where the install order is down to preference it may be possible to provide the right order through a boss/loot userlist provided by loot. OR step would have to tell users to deactivate auto-sorting for that mod and where to put it then.Long post short: Yes, MO would then move mods around based on plugin order. And it should not be necessary to turn this default behaviour off as it IS the usual behaviour for esp+bsa bundles when used outside MO. This IS the case the mod author has tested for and wrote his installation instructions for.

The documentation is present on each of the individual mod pages here on the forum. We do extensive testing in order to achieve a specific result (what we feel is the best result). Mod author's installation instructions/notes are always taken into account and is usually followed. I also, believe going forward we will not be recommending extraction of BSAs unless we're needing to hide assets within that BSA (unless you can come up with a way to do this without extraction ). Regardless I wasn't implying anything regarding BSA extraction. I just needed to know if the mods were going to be moved around with the plugins.

 

As for the reason STEP recommends a specific installation order, it is to achieve a specific result so that specific assets of one mod will overwrite assets of another mod (personal choices as you put it, but that how the STEP Guide works). This applies to the entire 2.F section of the guide as well as any mod in the Guide that isn't in alphabetical order. Those mods have been specifically place at their location in the Guide in order for specific conflict resolution. This has always been how STEP achieves it's results.

 

As to this new method's effect on the STEP Guide, that can not be discovered without testing the method out. It may or may not cause issues with section 2.F of the guide as well as the other mods that have been specifically placed. One thing I know it would cause issue with is the sections themselves. Gameplay mods would end up in the Animation section, Fixes mods could end up down in the Gameplay area, etc etc. This would force us to rethink and possibly reorder the entire STEP Guide because the sections would no longer stay together as a group. This is not a huge issue, but it will force us to rethink our management of mods and the Guide. Section N (coming in the next release), which needs to be loaded close to if not at the end of the plugin list, might have some issues as well.

 

 

@Tech

snip/

In short, as long as MO can proxy the behavior of Wrye Bash, we are good.

This... but WB's mod order and plugin order isn't tied together. They are independent from one another. (can't say for NNM but I'm pretty sure those are independent there as well). MO would be the only manager to tie these together. I'm not sure how the community as well as long-term users (who have no knowledge of this thread and discussion) would receive this new behavior.

 

I do see the merit in it, without a doubt!. I simply have some STEP management concerns with it and how things will fall into place. It's hard to tell how the management would work out at this point.

[TheBloke]

Regarding tying mod order (left pane)  to plugin order (right pane):  Same as Tech said, I have a number of concerns regarding this.

 

While it might simplify things, it does seem to risk being a backwards step.

 

Personally speaking, as long as it's possible to disable it and still achieve current MO behaviour, I would be fine.

 

But I do feel that MO's ability to achieve fine-grained mod prioritisation, even on a file-by-file basis (when hiding conflicting files), is one of its great strengths.  I extract all my BSAs, and pay close attention to the Conflicts tab of every mod.   I've followed a number of STEP guides, both in terms of coarse-grained ordering of mods, and fine-grained control of individual files by file hiding.  On top of that, I've done a lot of my own conflict resolution, in particular adjusting priorities and hiding files to ensure that conflicting scripts are applied in the order I think/know is best.

 

There is already a 'soft' link between plugins and mod ordering: the "Potential mod order problem" warning, which causes me to sometimes re-order my mods in the left pane to match their positions in the right pane.  To my mind, that is as much of a link as I would personally want to see.

 

As I say, as long as it's a change that can be disabled, it won't hurt me personally, so if it's really the best or only solution for the community at large, then fair enough.  But it does feel like a default removal of a useful ability, and I feel that the underlying problem that generated this thread could be fixed without needing to resort to that, if the "treat vanilla/DLC content like mods" solution I described above is workable (and I can't currently see why it wouldn't be.)

 

On top of which, it would seem a big shame if STEP guides had to go through a lot of convoluted work to update to the new feature, as Tech described; and they couldn't just tell users to turn off the auto-sorting because that would still leave the underlying Unofficial Patch problem.  So a solution to that problem that doesn't require auto-sorting, and thus doesn't invalidate every STEP guide, feels strongly preferable to me.  

[TheBloke]

The installation order is only relevant if there is a conflict between two mods, otherwise the order there simply doesn't matter.

Yes; but the number of those conflicts is not insignificant, I would say. 

This... but WB's mod order and plugin order isn't tied together. They are independent from one another. (can't say for NNM but I'm pretty sure those are independent there as well). MO would be the only manager to tie these together.

I have no knowledge of how WB sorts mods (only ever used it for Bash Patch), and I've not used NMM for more than a year (since I discovered MO and scrubbed NMM forever from my drive; Recycle Bin is too good for that crap.) So I don't have any knowledge of how these sort BSAs.What I do know though is that there are quite a number of mods whose installation instructions include the words "When asked whether to overwrite files, choose YES|NO". And/or: "For compatibility with ModX, install ModX first then install my mod and choose to Overwrite all files"I assume these are mostly scripts, but I believe there are some other kinds of files that can also sometimes be loose?So, with regard to: 

For the cases where two mods conflict and the order of the resolution is important (not down to preference), step should probably be very very careful proposing anything other than the plugin-order anyway. Think about it from the perspecive of the mod author: If he knows his mod contains scripts that conflict with scripts from other mods and thus may break the game HE will have to ensure users install his mod correctly.

 My experience is that many mod authors do include specific file-ordering instructions, and these could definitely differ from plugin order.Specifically, mod authors generally always assume NMM, and they give Overwrite instructions accordingly. Those instructions are separate to plugin ordering, where they will either say "use BOSS/LOOT", or they will give a specific order. But that's treated entirely separately from their instructions regarding any loose file conflicts.In other words, yes the mod author will know when he has conflicts and yes he will give instructions for prioritisation based on that: but he has no expectation of this being tied to the plugin order, at least in the case of scripts.So if we do suddenly have auto-sorting, I believe there's a definite risk of the defaults being out of line with the expectation and testing of many mod authors. And that's irrespective of whether a STEP guide is followed. (Indeed, a STEP user will be much better off than the average user, because the guide will be assuming MO where the mod author will not, and the whole purpose of the guide is to test and achieve multi-mod compatibility.)Without auto-sorting, any MO user who understands even the basics of how MO works could easily translate those NMM-specific instructions into MO mod-list ordering.My fear is that with auto-sorting, such a user is likely to just assume that the auto order 'must be right', when it may well not be - particularly with regard to scripts. And/or such users will start flooding the Nexus forums with "I use MO and it's sorted your mod ABOVE ModX, but you said I need to overwrite files from ModX, is that OK?" Which won't be good for anyone, least of all the reputation of MO (especially as apparently some prominent community members are already inclined to be down on it; to their shame.) Edited May 27, 2014 by TheBloke

[SkepticalJoker]

Tannin,

 

You're putting far too much stock into LOOT. Yes, you should always begin with LOOT in order to have an intelligently sorted starting place. Unfortunately, mods conflict in complex ways that LOOT can't always sort correctly, hence the user list. However, even the user list is inadequate to resolve certain conflicts that must be handled by manually placing plugins into specific positions in relation to other mods. The user list only lets you tell LOOT to sort mod x before or after mod y, but you can't tell it to place mod x into position x.

 

It seems to me you're proposing a regress. We would lose an extremely advantageous feature in favor of a highly restrictive plugin list. I'm no mod author or power modder, I think I represent typical users, and what you propose would ensure that I could never update MO. I would have to stick with the current version to avoid this new restriction. :( 

Edited May 27, 2014 by SkepticalJoker

[Neovalen]

I'm glad TheBloke and ScepticalJoker is here speaking my thoughts through ESP as I don't have the time to type out a huge response but I agree with them that the proposed method (linking install order to plugin order) is not a good one. One of MOs biggest strengths is the flexibility and control it gives to users while maintaining a clean data dir.

 

If TheBlokes method of auto detecting official DLC and turning them into "mods" is feasable it definitely gets my vote for the easiest to understand and work with from a guide writing and mod user perspective.

[EssArrBee]

I'm glad TheBloke and ScepticalJoker is here speaking my thoughts through ESP as I don't have the time to type out a huge response but I agree with them that the proposed method (linking install order to plugin order) is not a good one. One of MOs biggest strengths is the flexibility and control it gives to users while maintaining a clean data dir.If TheBlokes method of auto detecting official DLC and turning them into "mods" is feasable it definitely gets my vote for the easiest to understand and work with from a guide writing and mod user perspective.

I kinda agree too. Actually, I think that MO's current method is still the best even with the problem that the UPPs create. And that is really it, the UPPs limit our ability to have conflict resolution based on install order instead of the old way which is not optimal. MO already creates the best setup for data folder conflict resolution, since most of that is textures and meshes, divorcing the BSA load order from plugin order is the best method available. There just isn't any solution out there from a mod management perspective that is even close. Most users that don't like it are misinformed due to the lack of knowledge some have on the issue. I'd say we need to a better job informing people, and not try to change the whole system.

 

@Tannin, your current method is the best unless you somehow allow for BSA load order to be independent of install order and plugin order. Install order is the best option for conflict resolution, since files can just be hidden easily, so just tell everyone else to jump off a cliff and leave it the way you have it now. Everyone else will just have to figure out how to deal with the DLC and UPPs on their own which is a simple five minute solution of moving DLC to the mods directory or the UPPs to the data directory.

[fireundubh]

I think any solution that involves reengineering MO is too much.

 

False-flagging ESPs as ESMs is itself a workaround. It shouldn't even be possible. In fact, it is an unofficial third-party tool (read: hack) that makes it possible.

 

Instruct, or prompt, users to copy, move, or link the DLC in the mods folder so that the few false-flagged plugins that exist don't cause problems.

 

What has happened in these three threads is a lot of deep thought and proposed effort to climb a mountain that is actually a molehill.

[z929669]

It may be just me, but I don't see Tannin's proposed method as strictly tying asset prioritization (mod pane at left) to plugin prioritization (plugin tab in right pane). maybe this is basically true with regard to mods that have plugins, but most mods do not have plugins. For those that DO have plugins, we can decouple and retain current functionality (except without the confusing mechanics of the current Archive Tab functionality) ... Am I missing something fundamental though?

 

It would seem to me that Tannin's latest proposal would definitely make life simplest for the novice modder not following STEP or any other guide. It seems like a 'safe' approach to those that know little. Sure, LOOT or even BOSS should not be totally 'trusted' (the latter less so than the former), and I actually mentioned this a couple posts up as one potential problem; however, LOOT is under active dev and refinement, and we can expect that errors will be corrected (especially if our community gets better at submitting bug reports and change requests to BOTH USP and LOOT teams). For any inconsistencies in how we do things around here, we have classic MO functionality and a left-right pane decoupler. (oh and with WB and any other mod manager, installation prioritization is independent of plugin prioritization only with respect to loose files. BSA prioritization is still coupled to plugin priority. It's just that loose assets always win against plugin assets).

I think any solution that involves reengineering MO is too much.

False-flagging ESPs as ESMs is itself a workaround. It shouldn't even be possible. In fact, it is an unofficial third-party tool (read: hack) that makes it possible.

Instruct, or prompt, users to copy, move, or link the DLC in the mods folder so that the few false-flagged plugins that exist don't cause problems.

What has happened in these three threads is a lot of deep thought and proposed effort to climb a mountain that is actually a molehill.

Indeed. This or instruct them to copy USPs into /Data/ both seem to be the simplest solutions to the problem ... but Tannin has not cozied up to the idea and seems more intent on making fundamental changes ... so ... whatever he wants works for me as long as it works!

[TechAngel85]

Installing my test setup for 2.2.9 and using MO 1.2.1. Loot is working great but this brings up another issues with tying the mod list and the plugin list together. Why about mod which have mulitple ESP files (plugin) which Loot sorts in different locations. One of these mods that nearly every single modder will have is SMIM. For the STEP user, we'll intergrate those ESPs into the STEP Patches; however, what about everyone else? The mod can't be in five different locations at once. How would these mods be handled if the two lists are tied together?

[z929669]

Installing my test setup for 2.2.9 and using MO 1.2.1. Loot is working great but this brings up another issues with tying the mod list and the plugin list together. Why about mod which have mulitple ESP files (plugin) which Loot sorts in different locations. One of these mods that nearly every single modder will have is SMIM. For the STEP user, we'll intergrate those ESPs into the STEP Patches; however, what about everyone else? The mod can't be in five different locations at once. How would these mods be handled if the two lists are tied together

The main plugin should dictate, not the add-on plugins. For STEP, the mod would (should) not have any plugin reference and so would behave as a 'decoupled' mod I hope.

[Kuldebar]

fireundubh, on 27 May 2014 - 2:05 PM, said:

I think any solution that involves reengineering MO is too much.False-flagging ESPs as ESMs is itself a workaround. It shouldn't even be possible. In fact, it is an unofficial third-party tool (read: hack) that makes it possible.Instruct, or prompt, users to copy, move, or link the DLC in the mods folder so that the few false-flagged plugins that exist don't cause problems.What has happened in these three threads is a lot of deep thought and proposed effort to climb a mountain that is actually a molehill.

I think Fire is making a very valid point here. Compensating and compromise (and a lot of work on Tannin's part) in order to support previous (arguably) questionable compromises made by others can lead to the generational Xerox copy of problem solving, or as people used to say, "bailing wire and bubblegum". It's not that it won't work, or can't work; it's that it perpetuates a mess -a mess you yourself didn't make.

 

There needs to be a firewall break and it's best to keep logic intact and not just go along to get along. The lie down with dogs and wake up with fleas, also works as a descriptive image. I don't think I left out any pertinent metaphors. :)

[keithinhanoi]

False-flagging ESPs as ESMs is itself a workaround. It shouldn't even be possible. In fact, it is an unofficial third-party tool (read: hack) that makes it possible.

 

When I read this, I had to chuckle. With all due respect to everyone here - we are playing a modified game. A good number - possibly a majority - of mods in STEP are hacks of one sort or another.

 

SKSE? It injects itself into the executable to do things Skyrim wasn't designed to do. A hack. Sheson's memory block fix? A hack. ENB? Also a hack. 

 

All of these things "shouldn't even be possible." Now, let's just for argument's sake say that the UPs were made as .esm master plugins, and not false-flagged .esp plugins. There are plenty of very well known mods that use an .esm master plugin. Those get special status up (early loading) in the .esm master section of the load order. And the UPs as esm files could still require to be "sandwiched" between the official DLC .esm/.bsa content. And we'd still have the same thing to work out.

 

There's no need for me to say anything disparaging about the UPs because they do my game a whole heck of a lot more good than bad. Yes, they require somewhat peculiar requirements in terms of their load order (both plugin records & BSA encapsulated assets), but the decision to do so wasn't made easily and without thought - and it is what it is now - so please for the love of the divines can we move on?

 

Indeed. This or instruct them to copy USPs into <skyrim>/Data/ both seem to be the simplest solutions to the problem ... but Tannin has not cozied up to the idea and seems more intent on making fundamental changes ... so ... whatever he wants works for me as long as it works!

 

I'm afraid that the one fairly major drawback to copy the UPs BSAs into <Skyrim>/Data is that MO no longer "sees" those assets when reporting file conflicts.

 

So it seems the solution of making MO see the DLCs as mods in the left-pane priority list would be a better way to go, however the workaround is implemented.

 

This is not going to happen. You either let MO manage the mod manually then the BSAs are ordered in plugin order and loose files within a mod overwrite the bsas of that mod and all of lower priority or you manage manually then you specify BSA ordering for that whole mod and mod priority manually and loose files within a mod will still overwrite the bsas of that mod and all of lower priority.

In essence: For auto-sorted mods you only ever manage one list: plugin order. For user-sorted mods you manage three lists: plugin, archive, mod

 

Thanks so much for taking the time to answer my questions (or show how your concept resists the "mud" I was flinging at it!)

 

I have a much better picture of what you're proposing, but I'm still unclear how the manual user management of mod/BSA/asset priority would be handled GUI-wise.

 

Are you saying for user-prority managed mods that their BSAs could be manually drag-and-drop sorted in the Assets tab, -or- all of the mod's BSAs / loose files' priority would be set by the mod priority (drag-and-drop sorted as you can do now?)

 

If the answer is the latter, then my question is where does that mod get put priority-wise if I then turn on MOs priority sorting for it (based on plugin order) - if there are a bunch of user-priority sorted mods between the two plugin-order-sorted mods where the mod for which I'm toggling MO-priority-management to on should go?

 

Example - say at first I've got this mod priority order in the left-hand pane:

   A - priority set by plugin-order

   B - priority manually set by user

   C - priority manually set by user

   D - priority manually set by user

   E - priority manually set by user

   F - priority set by plugin-order

   G - priority manually set by user

 

Then I toggle the priority management for Mod G, which according to the plugin load order as set by LOOT, should go between Mod A and F.

 

So where would MO place Mod G?

 

And one other question. What would happen if a mod in the left-hand pane priority list contains multiple plugin & BSA pairs, which LOOT actually orders in quite "distant" plugin load order priority slots?

 

I can't think of any mods off the top of my head that do this, but certainly MO allows users to pile a bunch of different mod plugin/BSA pairs into one mod slot in MO's priority/"install" order list. 

 

Since in your concept the default is sort a mod's priority/"install" order based on its plugin load order (as set by LOOT/BOSS), then it's unclear where the mod would be placed when it contains multiple plugins in quite different load order slots.

 

Anyhow, if these kinds of catch-22 things have already been worked out - I think your idea is quite sound, because it starts off "safe", working in a very similar way to other managers, but adds the flexibility of still "mixing" loose assets to be effectively installed between plugin-order-sorted BSA assets (I'm not sure everyone else is catching that fact,) AND also allowing users to switch to full control over plugin/BSA/loose file load/priority order.

 

It really is a "have my cake and eat it too solution".

 

 

Finally - a parting note on the efficacy of LOOT:

 

LOOT doesn't really "know" the correct load order. It mainly bases the load order on two things - explicit masters (in a plugin header's master list) and plugin record conflicts. The goal is to make sure plugins load after their masters, and with the least number of plugin record conflicts. This is a very smart algorithm, and can address most conflict issues.

 

But it can't account for implied masters - plugins which are built with the assumption of a certain load order, but didn't require any masters to be added to its head, and especially important - LOOT can't on it's own account for required BSA asset load order.

 

That said, LOOT's own special masterlist is getting updated frequently, which is step-by-step growing with rules that handle these two areas that its algorithm can't handle - but it's going to take a while, and there are lots of mods out there that were built based on a BOSS load order which could have an implied master and/or assumed BSA asset load order situation.

 

What I'm trying to say here is that although having BSA priority order set according to accompanying plugin order is probably in most cases result in a stable game - it is still entirely possible that LOOT will produce a plugin load order that sets the BSA asset priority order with some undesired consequences or things that don't work as expected in-game.

 

So - my vote is still in with keeping user control over BSA asset priority order in some manner in MO

 

However, if it turns out that your concept is not entirely feasible, then I'd like to put in my vote for TheBloke's suggestion for handling the "exceptional" case of the UPs, although it would still leaves setting plugin load order tied to corresponding BSA asset priority order entirely in the hands of the user.

Edited May 28, 2014 by keithinhanoi

[TechAngel85]

Just out of pure curiosity, I did a comparison from the Fixes section of the STEP Guide with MO now vs the new method. Here what I came up with:

 

 

Mod List for MO Now  >>>>>>>>  Mod List for MO new method

 

Appropriately Attired Jarls....................Unofficial Skyrim PatchArgonian Decapitation Fix....................Unofficial Dawnguard PatchBowlegged Jump Fix............................Unofficial Hearthfire PatchBrawl Bugs Patch................................Unofficial Dragonborn PatchClothing and Clutter Fixes....................Skyrim Project OptimizationConsistent Older People.......................Appropriately Attired JarlsCultist Mask Decapitation Fix................Argonian Decapitation FixDead Body Collision.............................Bowlegged Jump FixDistant Decal Fix.................................Brawl Bugs PatchDouble Cursor Fix...............................Cultist Mask Decapitation FixFast Travel Timescale Fix......................Dead Body CollisionFuz Ro Doh........................................Guard Dialogue OverhaulGuard Dialogue Overhaul.....................Distant Decal FixRagdoll Paralysis.................................Double Cursor FixSkyrim -Elys- AltF4.............................Fuz Ro DohSkyrim Project Optimization.................Ragdoll ParalysisSmart Souls.......................................Skyrim -Elys- AltF4Trade And Barter - Hearthfire...............Consistent Older PeopleUnofficial Skyrim Patch.........................Smart SoulsUnofficial Dawnguard Patch...................Trade And Barter - HearthfireUnofficial Hearthfire Patch.....................Weapons and Armor FixesUnofficial Dragonborn Patch..................Clothing and Clutter FixesWeapons and Armor Fixes....................Realistic ForceRealistic Force.....................................Fast Travel Timescale FixXP32 Maximum Skeleton......................XP32 Maximum Skeleton

 

The new list is simply a best guess-timation by taking the mod list and rearranging it to plugin priority. Mods that had no plugin were simply placed in the list where they were installed. You can see how huge of a difference the lists are. Now imagine putting in all the STEP sections. Mods would be all over the place and not sticking to their sections as the Guide lays them out. I've finished through the Landscape section of the Guide and I have to admit that I'm not liking what I'm seeing if these two things (mod list and plugin list) were merged and tied to one another as the new method suggestions. (Strictly speaking from the viewpoint of what in the world would we do with the Guide to compensate).

[fireundubh]

When I read this, I had to chuckle. With all due respect to everyone here - we are playing a modified game. A good number - possibly a majority - of mods in STEP are hacks of one sort or another.

As far as I know, SKSE, Sheson's memory hack, and ENBSeries didn't need Tannin to reengineer MO to solve a few obscure problems with neck seams.