Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[keithinhanoi]

As far as I know, SKSE, Sheson's memory hack, and ENBSeries didn't need Tannin to reengineer MO to solve a few obscure problems with neck seams.

 

Sure, but MO has a history of changes being made to accommodate tools / mods / NMM updates that would otherwise not work due to the way MO handles mod management through its virtual file system.

 

Some viable suggestions have already been offered for a workaround with the current version of MO to the particular case of the UPs need to mix between the DLCs in priority order. In a "sandbox" type of environment as STEP provides, a specific guide to a workaround can be created, and consistent results can be expected.

 

However, I don't think Tannin's new suggestion of changing how MO handles priority management is just a knee-jerk response to the UP/DLC priority order quandry.

 

He's looking at the larger potential looming issue of mod authors creating mods with the assumption that the accompanying BSA assets will be loaded in the same order that their plugin is. This assumption is based on the way the game works without using any 3rd party mod manager, and it is also the assumption when using the mod manager that AFAIK the vast majority of people modding Skyrim (and probably other TES games) are using, NMM.

 

Right now, in order to adhere to that assumption in MO, the user needs to manually set BSA priority to match plugin order.

 

Tannin is talking about automating that as a default in MO, but still allowing for manual intervention by the user.

 

Yes, it would help address the special "hack" (or whatever) situation of the UPs that otherwise works correctly when not using MO, but it's more than just addressing that.

 

I could also fall back to the argument that all of these products of hundreds of hours of people's free time are shared with us at no cost, and it is up to us whether to use it or not. And on that note, I feel it is extremely gracious of Tannin to ask for input here. Nothing says that he is required to do so. 

Edited May 28, 2014 by keithinhanoi

[noobzor]

Forgive me if I missed this somewhere...I had 13 pages of posts to catch up on, and only about an hour to spare, so I may have missed this...

 

I saw somewhere Tannin saying that we shouldn't care how MO manages to prioritize BSA's based on install order instead of always being overwritten by loose files, and that an explanation would only make sense if you have a CS degree and lots of experience with computers. I do have a CS degree, and more than 10 years experience in software development, and I am very interested in understanding how MO accomplishes this. Not that I need to understand it, really, MO is just interesting. If this was covered already, can someone point me to it?

 

 

Also, if we leave the BSA un-extracted, do the file conflicts still show up in the "Conflicts" tab when looking at an installed mod's details, or do they only show up in the "Data" tab on the right-hand pane?

 

I am another user who always extracted BSA's, and I admit I did not thoroughly read the documentation and did not have a clear understanding of how the BSA prioritization worked in MO. I had always assumed that BSA's were always overwritten by loose files, regardless of the install order, and had no idea that files inside a BSA would show up in conflict resolution. That's what I get for not reading the documentation. Now, I know better.

 

That being said, this behavior is contrary to what even someone with advanced knowledge of Skyrim modding would expect if they were using MO for the first time. I prefer the power and flexibility that MO gives me to order things pretty much however I want (and consequences be damned!), but I can see that posing a significant entry barrier for the average user. MO will always be an advanced-user-only tool unless it has a simplified mode that more closely reflects what users are accustomed to from other mod managers.

Edited May 28, 2014 by z929669

[EssArrBee]

Also, if we leave the BSA un-extracted, do the file conflicts still show up in the "Conflicts" tab when looking at an installed mod's details, or do they only show up in the "Data" tab on the right-hand pane?

If you tick the BSAs in the archive tab then you can see the conflicts, otherwise it doesn't dhow because loose files always win.

Forgive me if I missed this somewhere...I had 13 pages of posts to catch up on, and only about an hour to spare, so I may have missed this...

 

I saw somewhere Tannin saying that we shouldn't care how MO manages to prioritize BSA's based on install order instead of always being overwritten by loose files, and that an explanation would only make sense if you have a CS degree and lots of experience with computers.  I do have a CS degree, and more than 10 years experience in software development, and I am very interested in understanding how MO accomplishes this.  Not that I need to understand it, really, MO is just interesting.  If this was covered already, can someone point me to it?

The source code is available on sourceforge I believe.

[Tannin]

Sorry mates, can't answer everything now.@TheBloke: I understood your proposal. Basically the "mods" Skyrim, Dawnguard, Hearthfire, Dragonborn wouldn't physically contain the esp+bsa but they would represent the files in the data directory.I like this proposal, it lets us fixes the problem at hand with manageable changes to MO, the sole reason I haven't agreed to go this route is that it isn't as automatic as it could be. Basically you would install mods as usual and the PMOP warning is guaranteed to show up for the unofficial patches and the game will only work perfectly if the user reads and follows the specified suggestions therein.Unfortunately in the past people have advised users to ignore the PMOP warning and thanks to how the internet works this can not be undone. We will always have users stumble over those suggestions and go "but people have told me to ignore the warning"A question to the informed STEP staff: The step guide currently suggests to prefer loose file versions of mods and unpack the reminaing BSA.What happens if a user installs STEP with NMM and ignores the suggestion, that is: he prefers BSA versions and doesn't unpack?Will his game break, exhibit glitches or are the differences cosmetic in nature?Say we did a compromise of my last suggestion: MO by default activates auto-sorting for mods containing bsas but not for loose-file mods (You could of course always toggle it)In this case, by default asset ordering in MO would work almost exactly like in NMM (except that BSAs would still be able to override loose files if their priority is higher)Would that be feasible for step? The advise could still be to unpack specific BSAs (where step knows a better order than the plugin-order) and auto-sorting wouldn't affect the mod.Or you could tell users that - instead of unpacking bsas - they could disable auto-sorting for that mod which would have the same effect while being easier to revert.I want to assure you guys that I won't implement a solution that breaks current step-guides, my primary gole is to reduce problems for as many users as possible. This includes both step users and users that install mods without a guide.But the latter group will not have a tried&tested installation order to follow. They install the mod, either as a esp+bsa bundle and rely on loot to sort it correctly or they follow the mod authors instructions which will be written with NMM in mind. Ideally they would largely apply to MO as well.That's why my solutions may seem more complicated than necessary: I'm not only thinking about how to solve the immediate problem that mod-bsas can't be loaded interleaved with official bsas, if I'm going to implement something complicated why not kill two birds with one stone.

Why about mod which have mulitple ESP files (plugin) which Loot sorts in different locations.

For BSA mods this is not a problem, because esp bsa is a one-to-one mapping. Each bsa would be in the right place as specified by the plugin-order (which is the same as NMM would have placed them which is the same as what the mod author probably tested) and the mod order is irrelevant. MO would still have to place it somewhere, maybe aligned with the largest esp (in terms of filesize) or the first esp when ordering alphabetically. It's arbitrary but that doesn't matter, because, again, the mod order has no impact.For loose file mods MO would probably apply the same mod ordering (i.e. by largest esp). This is still arbitrary but less so than installation order! Unless you have a guide telling you where to place a mod this solution is still better than having mods simply appended to the end of the list and if you have a guide all you have to do is toggle auto-sort and do as the guide tells you.

[Garfink]

@Tannin

 

Just to keep things in perspective, and has to be said, you're awesome and MO is the best.  Skyrim Modding would be so much of a chore without your mod.  Thank you.

[DocClox]

I want to assure you guys that I won't implement a solution that breaks current step-guides, my primary gole is to reduce problems for as many users as possible. This includes both step users and users that install mods without a guide.

I have to say, I like your design priorities. That said, if it comes down to making things easier for STEP or easier for the more general case, I'd say go for the more general one. Any group that expects its users to understand ddsopt shouldn't have a problem relocating three ESP/BSA pairs :)Of course, I'm new around here so I may not reflect the orthodox STEP viewpoint :)

[keithinhanoi]

Tannin, your "compromise" version of your suggestion sounds great. So that starts the MO user off with everything working as it would with no mod manager / NMM, and then the user is "free" to start changing things to override that paradigm.The main matters then are technical implementation, interface / GUI implementation, and also compatibility with the other games that MO works with besides Skyrim (which I've need seen any mention of yet.) 

 

Why about mod which have mulitple ESP files (plugin) which Loot sorts in different locations.

For BSA mods this is not a problem, because esp <-> bsa is a one-to-one mapping. Each bsa would be in the right place as specified by the plugin-order (which is the same as NMM would have placed them which is the same as what the mod author probably tested) and the mod order is irrelevant. MO would still have to place it somewhere, maybe aligned with the largest esp (in terms of filesize) or the first esp when ordering alphabetically. It's arbitrary but that doesn't matter, because, again, the mod order has no impact.For loose file mods MO would probably apply the same mod ordering (i.e. by largest esp). This is still arbitrary but less so than installation order! Unless you have a guide telling you where to place a mod this solution is still better than having mods simply appended to the end of the list and if you have a guide all you have to do is toggle auto-sort and do as the guide tells you.

 

Apologies because I also asked this question after TechAngel85 - I think I read it wrong at first!

 

So from what your describing, it will be very important for users to pay close attention to what's displayed in the Assets tab - because that's the only place that this one-to-one mapping would be visually represented.

 

If set up this way it would also means that it would be very important to add another column in the right-hand mod priority/"install" list pane with clear visual indication of whether a mod's assets are plugin-order priority managed, or manually user-managed.

[phryxolydian]

Install order with Opted Textures and all BSAs checked in the archive tab:

• Skyrim Opted Textures
• USKP
• Dawnguard
• Dawguard Opted Textures
• UDGP
• Hearthfire
• Hearthfire Opted Textures
• UHFP
• Dragonborn
• Dragonborn Opted Textures
• UDBP
• HRDLC1 (these three could be the optimized versions repacked without plugins)
• HRDLC2
• HRDLC3
• UHRP (untick the plugin, tick the BSA)
• All other mods...

What the heck! When I followed the optimization guide I ended up with four folders: Vanilla Optimized Textures (which includes the DLCs), HRDLC1, HRDLC2 and HRDLC3. Did I mess it up somewhere? My MO looks like this:

 

 

 

[TheBloke]

@TheBloke: I understood your proposal. Basically the "mods" Skyrim, Dawnguard, Hearthfire, Dragonborn wouldn't physically contain the esp+bsa but they would represent the files in the data directory.

I like this proposal, it lets us fixes the problem at hand with manageable changes to MO, the sole reason I haven't agreed to go this route is that it isn't as automatic as it could be. Basically you would install mods as usual and the PMOP warning is guaranteed to show up for the unofficial patches and the game will only work perfectly if the user reads and follows the specified suggestions therein.

Unfortunately in the past people have advised users to ignore the PMOP warning and thanks to how the internet works this can not be undone. We will always have users stumble over those suggestions and go "but people have told me to ignore the warning"

Thanks for the feedback!

 

I was going to mention this already yesterday but didn't want to add more walls of text: what about, in addition to the auto-management of Skyrim + DLCs I mention, also have automatic handling for Unofficial patches?

 

EDIT:  I've just been re-reading some of the earlier posts, and I realise now that you did already address 'treating Unofficial patches as special', and didn't like the idea.  I do understand why (there might be others who want to be special in future.)  Nonetheless, I do still feel the following is a workable solution.  There won't ever be more official DLCs for Skyrim, and therefore there shouldn't need to be more unofficial patches.  If there are, then, well, a line could be drawn - and, more to the point, if a future 'special' case comes along, hopefully at least it will be created with some deference to MO, which after all is now the official platform of STEP, which is not a small body of support.

 

Also, I wonder if it could be made configurable.  Not hardcoded, but a text config file that defines what makes the DLCs special (list of files to look for in Data) and what makes the Unofficial patches special (Nexus IDs, filenames, or false-flag ESM as others mention after my post.)  That way, it can be updated in future without further development work.

 

Finally, I would be glad to volunteer with coding assistance for this, or any other solution you decide to implement.  My C and C++ is a bit rusty, but I should be able to help at least with some of the simpler but time-consuming tasks.  (Or I can help far more if any of this can be done via Python plugins?)

 

TLDR, brief summary: Extend my proposed auto-detect handling of vanilla + DLC to contain one extra 'special mod type', namely the unofficial patches. Then have automatic, default sorting for all of vanilla, DLC and unofficial patch content. If necessary, disable re-prioritisation of these mods, and provide smart enable/disable such that if Dawnguard is unticked, it auto unticks UDGP.

 

In detail:

 

1. New user installs MO. He gets a popup "MO detected the following content, and has created mod list entries for it: Skyrim, Hearthfire, Dawnguard"

 

2. He then gets a second popup: "Please now download the Unofficial Patches for Skyrim, Hearthfire and Dawnguard. Using Skyrim mods without these unofficial patches is not supported by most mod authors, and strongly discouraged. Please click the following link to find download locations for these mods: <link to new or existing STEP wiki page which talks about Unofficial patches and has clear links to each of them>"

 

3. New special rules processing for installation of new mods: auto-detect Unofficial patches. Based on either: Nexus ID=xxxxx (e.g., 19 for UKSP, 23491 for UDGP), OR if the mod contains "Unofficial Skyrim Patch.esp and bsa".

 

If a mod is installed that matches EITHER of these rules (both rules checked in case Nexus is not used), then the mod is considered to be the appropriate Unofficial patch, and is handled specially

 

4. Special rules for vanilla, DLC content, and for Unofficial patches: They are always auto-sorted to the top of the mod list, in the interleaved order I defined before:

 

0 | Skyrim

1 | Unofficial Skyrim Patch

2 | Hearthfire

3 | Unofficial Hearthfire Patch

4 | Dawnguard

5 | Unofficial Dawnguard Patch

... etc

 

This time I have put them all in bold - they are now all special mods, where before only Skyrim/Hearthfire/Dawnguard/(Dragonborn) were special mods.

 

-----------

 

OR,  a modification to the above:

 

Have auto-detected vanilla/DLC content (as per my original proposal), and have auto-detected Unofficial content (as mentioned above), BUT - then merge them together. I.e. the user gets a popup telling him/her to install the unofficial patches. When he/she does, MO automatically inserts it into the appropriate vanilla/DLC mod, and then changes the name from "Skyrim" -> "Skyrim (including Unofficial Patch)"; "Dawnguard" -> "Dawnguard (including Unofficial Patch)".

 

Therefore the user has no choice as to how to sort the Unofficial patches, and no choice as to whether to disable them (once he's installed them.)  They're just considered part of the vanilla content, and MO handles this automatically. (at least unless he disables this auto management in the options.)

 

That would seem closest to how the Unofficial guys want it to be done, while making the least significant changes to MO's sorting/mod management paradigms.

 

To be honest, if I were doing it I might even go the whole way - and have MO automatically download and install all the unofficial patches (that it has detected it needs based on installed DLC) the first time it's installed.  But I don't know if that would be something you'd want to do.

 

-----------

 

Sorry to keep banging on about my idea, but in particular now you've said there's no technical problems with it, I still do feel it keeps closest to MO's current flexibility and paradigms, whilst adding new transparency for users regarding default sorting, and also enabling easy per-profile management of DLC content.

 

I will give up on the idea if you don't like these addendums :) But I had been going to mention that anyway so I thought I'd put it out there.

 

 

 

Thanks again!

Edited May 28, 2014 by TheBloke

[fireundubh]

Instead of checking specifically for the unofficial patches, Tannin could auto-detect ESPs flagged as ESMs by reading the ESP header in hex and checking if the ninth byte (0x08) is 01 (ESM).

Edited May 28, 2014 by fireundubh

[DocClox]

Instead of checking specifically for the unofficial patches, Tannin could auto-detect ESPs flagged as ESMs by reading the ESP header in hex and checking if the ninth byte (0x08) is 01 (ESM).

Sounds good. Many suggestions have relied on having a list of mods that needed special consideration. This avoids the necessity of maintaining an up-to-date list as more mods adopt the technique.

[TechAngel85]

Sorry mates, can't answer everything now.

 

snip/

 

A question to the informed STEP staff: The step guide currently suggests to prefer loose file versions of mods and unpack the reminaing BSA.

What happens if a user installs STEP with NMM and ignores the suggestion, that is: he prefers BSA versions and doesn't unpack?

Will his game break, exhibit glitches or are the differences cosmetic in nature?

First off, we specifically tell users that they can't use NMM with the STEP in the STEP Guide. They have to use MO or WB and preferably MO. NMM is not powerful enough to handle the complexity of STEP; therefore, I say that point would be moot from a NMM perspective because we'd be instructing them to switch to MO. From the 2.2.9 Guide:

"STEP now officially supports only Mod Organizer for reasons stated in the guide introduction. Although Wrye Bash is a mod manager, STEP recommends that it only be used as a helper application to Mod Organizer exclusively for the functionality of the Bashed Patch. The reason for the limited support of Wrye Bash is due to simplicity of this guide. Users who wish to use Wrye Bash can easily use that mod manager to install and maintain STEP. Wrye Bash users can use the Wrye Bash Guide for detailed instructions on setting up that exceedingly excellent modding utility package."

 

Second, several of the Team is currently testing STEP in a non-extracted BSA setup. One of the main reasons, we recommended BSS extraction came from the WB days. It is more necessary when using WB. With MO, this is not the case so that recommendation will likely be removed. The ONLY time we're going to recommend BSA extraction will most likely be for mods that need assets hidden so that a lower priority asset can win the conflict.

 

Say we did a compromise of my last suggestion: MO by default activates auto-sorting for mods containing bsas but not for loose-file mods (You could of course always toggle it)

In this case, by default asset ordering in MO would work almost exactly like in NMM (except that BSAs would still be able to override loose files if their priority is higher)

Would that be feasible for step? The advise could still be to unpack specific BSAs (where step knows a better order than the plugin-order) and auto-sorting wouldn't affect the mod.

Or you could tell users that - instead of unpacking bsas - they could disable auto-sorting for that mod which would have the same effect while being easier to revert.

The fact is, some of the auto sorted mod will still not follow the sections of the guide. From my current LOOT sorted setup, I would end up with Landscape mods in Fixes and Fixes mods in Gameplay. Undoing the  Auto-sorting on all of these mods would fix this issue but cause another one. If the mod location is tied to the plugin location, then manually sorting any mods would mess up the LOOT sorted plugins (because if you move the mod, you move the plugin). The only way this would work for STEP is if the bonded relationship between mod location and plugin location could be broken when toggling the auto-sort off for those mods. That way we could move the mod's installation location around while the plugin stays where it's suppose to be (as it is now in MO). This feature of breaking that relationship is essential and has been brought up by several users, not just Staff.

 

Many seem to be against this whole idea of tying the two together though. I suspect there will be a lot of "before running LOOT select all mods, toggle auto-sorting off, now run LOOT, now move mod XYZ to this location, etc etc". That why MO will function mainly as it is functioning now. As for STEP, we'd have to come up with some workarounds on the Guide due to all the mods that would be moved around by the auto-sort, but it could work.

 

Personally, I agree with the others. This isn't the best solution from the user viewpoint. This would mainly be helpful for no-nothing beginners just getting into the modding world, which imo is the minority group of your user base. Typically one would do what is best towards the majority group and the minority would simply have to learn. STEP is as much as about teaching newbie modders how to mod and use tools as it is for advanced modders who simply want a starting point. I would personally rather see the DLCs be added into the mod list (in whatever fashion) which would solve this entire problem and then drop the auto-sort idea. I know you're not exactly for that solution because the auto-sort would solve the issues from the PMOP; however, the majority here have voiced this to be the best solution from the modder viewpoint.

 

I want to assure you guys that I won't implement a solution that breaks current step-guides, my primary gole is to reduce problems for as many users as possible. This includes both step users and users that install mods without a guide.

But the latter group will not have a tried&tested installation order to follow. They install the mod, either as a esp+bsa bundle and rely on loot to sort it correctly or they follow the mod authors instructions which will be written with NMM in mind. Ideally they would largely apply to MO as well.

That's why my solutions may seem more complicated than necessary: I'm not only thinking about how to solve the immediate problem that mod-bsas can't be loaded interleaved with official bsas, if I'm going to implement something complicated why not kill two birds with one stone.

Any feature that will automatically move the mods around could potentially break the STEP Guides because STEP is based on install order (not plugin order) for conflict resolution reasons. Plugin order has always been handled independently from install/mod order via BOSS (now LOOT) and have never before been tied together. Anything that will take away or alter an install order priority will cause some issues for the Guides, as I've explained above and more below. With that said, I do understand your reasoning for the solution you are proposing and if it's implemented, STEP will figure out a way to work around it. Though the very structure of the Guide may have to change to not be so heavily dependent on install order. There hasn't been any discussion on this by the Staff yet.

 

For BSA mods this is not a problem, because esp bsa is a one-to-one mapping. Each bsa would be in the right place as specified by the plugin-order (which is the same as NMM would have placed them which is the same as what the mod author probably tested) and the mod order is irrelevant.

Here is where you are mistaken because mod order is completely relevant, especially for STEP! Take SMIM for example. It's not a BSA but some 5 (I think) ESPs. If the mod is placed where the first ESP is sorted, that might be okay, but it might not be either. There are many mods that are SMIM compatible (mainly textures). A user may want to overwrite the SMIM asset with the SMIM compatible mod (which also contains a auto-sort plugin) of his/her choice because they prefer that style over the SMIM style; however, if the SMIM loose assets are loaded after the users choice assets, because the auto-sort feature put it there, then you'll be breaking asset priority that was intentionally set by the user. This forces the user to manually sort these mods (how many newbies are going to know how to resolve these conflicts?). Again, the manual sort would have to break the mod/plugin location or the LOOT order would be thrown off by these manual sorts. Mod order is very important in modding!

 

Sorry about the book...but I hope I covered everything clearly.

[z929669]

 

... snip/

where does that mod get put priority-wise if I then turn on MOs priority sorting for it (based on plugin order) - if there are a bunch of user-priority sorted mods between the two plugin-order-sorted mods where the mod for which I'm toggling MO-priority-management to on should go?

 

Example - say at first I've got this mod priority order in the left-hand pane:

   A - priority set by plugin-order

   B - priority manually set by user

   C - priority manually set by user

   D - priority manually set by user

   E - priority manually set by user

   F - priority set by plugin-order

   G - priority manually set by user

 

Then I toggle the priority management for Mod G, which according to the plugin load order as set by LOOT, should go between Mod A and F.

 

So where would MO place Mod G?

 

/snip ...

Good question, and I would propose that (if implemented in whatever form) the plugin prioritization list SHOULD NOT translate to the mod prioritization list. Rather highlight (or gray out) the text or background of mods ...

• ... in the mod prioritization list that are prioritized by plugin list, and
• ... do the same for plugins in the plugin list that have content that is managed by the mod prioritization.

---

@Tannin

Thanks for being accommodating to STEP and all users

 

I like TheBloke's latest ideas about MO 'special' handling of vanilla and USP content with autodetection, auto-sort and text indicators (first option, not the second, less customizeable rendition)

 

I agree with Tech that STEP does not support NMM as a mod manager, although NMM is a proxy for Wrye Bash methodology and should be assumed to follow the same basic principles of WB. (I am currently assembling my thoughts about Tech's concerns and Tannin's revised, "compromise" proposal, so more from me later).

[TheBloke]

Hey guys,

 

Regarding the following edit on z9's OP:

 

• [*]Incentivizes mod authors to provide BSA 'hotfixes' as loose files (

NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)

 

FYI, we've been discussing this over in the REGS thread, and I confirmed that the problem is fixed in 1.2.1beta whenever one installs a new mod (or Replaces an existing mod).  Tannin has changed the BSA extraction such that it won't overwrite any existing loose files.

 

But I've found that the fix for this actually causes a new, more serious issue:  whenever an update patch is applied (example: upgrading Interesting NPCs from 3.05 to 3.06 using the 3.05->3.06 incremental patch archive), and that patch contains a BSA, any files in the patch BSA will not be extracted if they already existed in the original mod.  

 

So patch updates will be broken for any mods that use incremental patches, where the patch contains a BSA file that is meant to update files in the original release, and where the user has chosen to Extract all BSAs.  The "don't extract from BSA when existing file is found" rule is applying always, and that is not desirable in the case where the user is choosing to Merge an update into an existing mod, and is extracting all BSAs.  

 

More information including steps to avoid, in my post on the REGS forum here.

 

I've raised a bug report for Tannin, including a suggested way to handle the problem that solves both the original problem (dupe files between BSA vs loose, in a single archive) and the new problem:  https://issue.tannin.eu/tbg/modorganizer/issues/644

 

I thought I should mention it here so z9 can update the OP, and so people know that upgrading to 1.2.1 beta to fix the occasional problem of dupes between BSA and loose could lead to a more serious issue, for the time being until it's fixed in a new beta or full release.

[fireundubh]

@TechAngel85: I don't see why install order matters at all. You know you can drag mods in the middle pane to reorder them, right?