Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[DoubleYou]

What I need to know and am not 100% sure on is something I mentioned earlier. Will a ticked BSA be overwritten by an unticked, plugin-loaded BSA in Mod Organizer? That is the one thing I don't know. Could somebody do a simple test? It would be optimal if I'm wrong about that. If a ticked BSA can overwrite an unticked, plugin-loaded BSA, then it is a simple matter of unticking the Unofficial patches and DLC BSAs in the Archives tab, no re-arranging whatsoever.

 

As to my potential problems list, I'm merely listing all the files that are subverted with Mod Organizer's ticked BSA behavior that could potentially be issues, which coincidentally are all the files overwritten by the DLCs from patches loaded earlier. The ramifications outside Mod Organizer are limited only to fixes that potentially could have been overwritten by the DLCs after the switch to false-flagged esp ordering and are most likely non-issues for the Unofficial Patch team, but still worth investigating.

[Kelmych]

I have related but slightly different questions. How does MO handle a mod with a BSA but no plugin, or a mod with a BSA and a plugin but the plugin is unchecked in the plugins tab. I expect MO can provide the BSA to Skyrim with the VFS, but what order does this BSA get in Skyrim? If I have two mods each with a plugin and an unextracted BSA, and if mod A has lower installation priority in MO but the associated plugin is higher in mod order, which BSA would have higher priority when loaded into Skyrim?

 

By the way, with Wrye Bash the answer to the first question would be that the BSA is loaded into Data but is not loaded into Skyrim for either alternative since there is no associated active plugin. For the second question with WB the answer is that BSA order in Skyrim depends only on mod load order (plugin order).

[keithinhanoi]

Finally trying to catch up with this thread. Just want to say off the batm it's very difficult to follow due to all the misunderstandings and inconsistent use of terms.

 

 

So the new standard will have us all placing the Official DLC's in the MO's folder system?

 

 

 

That's funny - because I changed to this about a month ago, when I noticed that although I had my cleaned Beth's .ESMs in the correct priority/"install" order with the corresponding UPs, the BSAs were not loading in the proper sequence. To be crystal clear about this, some pics would help.

 

So this was my mod priority/"install" order in MO's left-hand pane:

Note that the (cleaned) Bethesda mods were only the .ESM, without their corresponding .BSA files.

 

But then, in MO's Archive tab in the right-hand pane, the BSA assets priority order shown was this:

 

 

 

This was of course despite having the correct plugin load order, as set by BOSS or LOOT:

 

 

 

So I figured I had two choices - either:

 

1. un-tick/uncheck just the official & unofficial BSAs so that MO wouldn't manage them, allowing them to load following the order of the .ESMs, or

2. duplicate or move the official BSAs into MO's mod folders for the three official DLCs (with (cleaned) in the name in first pic above), which would then allow MO to manage BSA asset priority 

 

I decided against #1, because even though the official & unofficial patch BSA's would get loaded following the .ESM plugin order, and theoretically those .BSA assets would get loaded before all of the ticket/managed .BSA and loose file assets, I recall asking Tannin about mixing & matching managed & not-managed .BSAs in MO, and he recommended against it.

 

I should also note with regards to option #1 that Update.bsa does not show up anywhere in MO's Archives tab, so there's actually no way to disable MO's management of that BSA, and to be honest I'm still unclear about how the load priority of official .BSA assets are handled by MO (maybe Tannin can clarify on this specific point.)

 

So, because I have an SSD where space is costly and "precious", and I have Steam's automatic updating for Skyrim turned off, I went with #2. I moved the BSA files for the three DLCs into the MO mod folders that already had the cleaned .ESMs, and stuck the original official .ESM's in /optional/original plugin, like this:

 

 

 

So, after moving all three of the official DLC .ESM & .BSAs in this way, then MO's Archive tab showed the .BSA priority load order matching my mod priority/"install" order - as can be seen here:

 

 

Note the lack of Update.bsa in this list, but my assumption is it is loaded after Skyrim - VoicesExtra.bsa.

 

 

At this point, as long as I leave all .BSAs ticked and managed by MO, I believe that theoretically I could unpack the unofficial patch BSAs and everything would work as expected - because MO treats all managed .BSA assets and loose files as "equals" in determining what gets loaded into its virtual file system with overwrite priority based on the mod priority/"install" order. Theoretically, as I said, but I have absolutely no reason to do it, so I'm not even trying it.

 

So what's the moral of the story here?

 

Well, if there was some confirmation on exactly how un-ticking/un-checking the Beth's & unofficial patch .BSAs would be handled by MO, then my option #1, as explained above, could be a viable solution for an update to the STEP Guide.

 

However it would require the .BSA assets priority loading in MO to work like this:

• Registered Skyrim base .BSA files load
• Update.bsa loads
• Un-ticked/un-checked .BSAs load following .esm / .esp plugin load order
• All managed (ticked/checked) .BSAs and loose files load, according to MO's mod priority/"install" order

If that's the way MO works now, then this whole sub-topic can hopefully be put to rest.

 

If not, then perhaps the solution could an option in MO to allow Skyrim's official DLCs to be treated as mods in MO's left-hand mod priority/"install" order pane - without having to copy or duplicate their files into the MO folder structure.

 

Otherwise, then yes, you'd have to do what I've outlined here (and others have already suggested.)

 

Major apologies for arriving "late" to the party and also for not posting about what I had done before now!

Edited May 26, 2014 by keithinhanoi

[Tannin]

@Tannin, DoubleYou & GSDfan

FOR THE RECORD: Does MO treat the DLC BSAs differently than USP BSAs with respect to the effect of checking or not checking in the Archives Tab?

Nooo, MO doesn't treat the dlc special. It also doesn't treat the grayed out bsas special.

The grayed out bsas are only grayed out because they are listed in the ini file and are therefore probably required by the game.

 

The problem is, again, actually quite simple and not caused by MO doing anything special: MO is based around the assumption that official game content (everything in the data directory) is loaded first and mods work as overlays to that. The mod isolation feature enforces this, it ensures that each mod is a separate layer on top of the game (both physically as well as logically) NOT mixed with the original content. That's what it's all about and this is how I developed features (hence controlling BSA load order made sense at the time I developed it)

 

And this assumption was valid for years for four different games until the unofficial patches decided to be loaded interleaved with the official content.

 

MO enforces that mod assets are loaded after the dlcs whereas loot (and the suggestions from the unofficial patches) make you load the plugins interleaved with the official stuff. the unofficial plugins have pretty much "elevated" themselves to the same level as the official content.

 

It's not MO doing anything special, its the unofficial patches that want to be treated different than other mods. These three mods effectively break a logical order that has been applicable since Oblivion (probably even Morrowind)

 

I'd also like to point out that for all the ranting Arthmoor has done agains MO he has never contacted me about his concerns. I have never had a bug report that MO might be causing this kind of problems. No one has contacted me with constructive feedback on the topic.

As someone who hasn't played Skyrim in over a year I don't stumble about these problems myself anymore.

So if people in the know had actually talked to me instead of ranting in secret this could have been solved a long time ago.

MO has existed before the unofficial patches were even started.

It has existed when the unofficial patches teams decided to make them fake esms that need to be installed between official stuff. THEY could have determined this causes problems with MO months before I ever had the chance. THEY could have made me aware of the problem way before releasing and given me a chance to solve it before it ever affected anyone. Instead they release, then rant, then wait for users to propagate the issue to me.

THIS is why the problem isn't fixed yet, NOT because I did anything extraordinary.

 

 

Now this was my pointless rant, how was it? Constructive much?

 

 

The best solution I can suggest now is: Don't unpack the unofficial patch bsas and don't tick their bsas on the archives tab! Ignore the warning sign (it will be removed in the next MO release).

The next version of MO will also no longer suggest users tick the archives by default, it will make it clear that this is only necessary (and reasonable) if you actually want to take control of the asset load order of that mod.

 

This, combined with the fact that BSA extraction will by default be disabled will hopefully reduce the number of problems for beginning users, now it's mostly important that we all don't advertise those features to users unless they actually understand what they're about.

[Kuldebar]

Tannin, I found that most enlightening, thanks.

[wolverine2710]

Tannin, thanks for the explanation and for giving a solution for the current situation. If I've interpreted your post correctly there IS a real solution possible.  I've read three things:

• LOOT/BOSS interleaving the unofficial patches. I think/hope by contacting them this can be changed in the masterlist.yaml file.
• The unofficial patches that want to be treated different than other mods
• Mod Organizer.

How would in your opinion a REAL solution look like?

Would this involve having Arthmoor (and team) changing things in the official patches or is there another real solution possible (MO changes)?

I hope that you and Artmoor can work something out. I also hope that Arthmoor responds in this thread to tell us what is possible or not..

 

Should I've misinterpreted things and making things  more complicated then I apologize upfront.

[Nearox]

Some time ago I was still crashing regularly until I moved those BSA into a mod folder so I could determine the load order. I had always thought this was some kind of silly oversight on my part. If only I had known back then, I would've reported it in...Thanks for the clarification Tannin!@ Tannin/Wolverine regarding

The unofficial patches that want to be treated different than other mods

As a non-programmer I would think that a solution could be to hardcode the load order of the unofficial patches and the official DLC BSA in MO, just like the vanilla BSAs seem to be hardcoded as well (at least their order never changes it seems). Edited May 26, 2014 by Nearox

[GSDFan]

What I need to know and am not 100% sure on is something I mentioned earlier. Will a ticked BSA be overwritten by an unticked, plugin-loaded BSA in Mod Organizer? That is the one thing I don't know. Could somebody do a simple test?

 

Oddly enough when I tried to do this the unchecked BSA would revert back to checked state when switching from the archive tab and back. It would also do it when double clicking any mod and closing it and with using the refresh button or exiting and restarting MO.

 

Is this normal behavior or do I need to reinstall MO? By the way MO version 1.2.1.

[fireundubh]

How would in your opinion a REAL solution look like?

A real solution would involve Bethesda Softworks fixing their software, but you can wish in one hand... 

Would this involve having Arthmoor (and team) changing things in the official patches?

They can't do anything. They have to false flag the unofficial patches for reasons already mentioned. 

Or is there another real solution possible (MO changes)?

The simplest, most elegant, and most reasonable solution is for users to put in the work to move, copy, or link the DLC as mods.Sometimes you have to configure software and the operating environment to work the way you want. Edited May 26, 2014 by fireundubh

[Tannin]

Tannin, thanks for the explanation and for giving a solution for the current situation. If I've interpreted your post correctly there IS a real solution possible.  I've read three things:

[*]LOOT/BOSS interleaving the unofficial patches. I think/hope by contacting them this can be changed in the masterlist.yaml file.

[*]The unofficial patches that want to be treated different than other mods

[*]Mod Organizer.

How would in your opinion a REAL solution look like?

Would this involve having Arthmoor (and team) changing things in the official patches or is there another real solution possible (MO changes)?

I hope that you and Artmoor can work something out. I also hope that Arthmoor responds in this thread to tell us what is possible or not..

 

Should I've misinterpreted things and making things  more complicated then I apologize upfront.

 

There are multiple solutions, don't know which is more real.

1. Have MO not apply its "magic" on the unofficial patches, thus having them load like the unofficial patch team intended. This is the simplest solution for everyone involved, only drawback is that it breaks the clean layering of mods in MO, at least for these mods

2. Remove the "magic" regarding BSAs from MO completely. This is a radical step as it will invalidate a lot of documentation we have and it would mean that MO users would again have to concern themselves with plugin bsas vs. loose files vs registered bsas

3. We could write a MO plugin that extracts the dlcs from the data directory and puts them into individual mods, thus automating the step proposed further up in this guide.

4. The unofficial patch team could release different variants of their plugins depending on the set of installed dlcs.

 

re 4.: My understanding is this, please correct me if I'm wrong:

The load order (as an example)

Skyrim.esm

USKP.esp

Dawnguard.esm

 

is necessary because USKP contains script fixes for the vanilla game that are not compatible with Dawnguard so they need to be replaced by the dawnguard content. The only reason USKP needs to be loaded before DLC stuff is because the DLCs need to be able to overwrite stuff from unofficial patches, correct?

 

Now if instead of releasing one patch for skyrim, one for dawnguard, one for hearthfire and one for dragonborn

they could release one patch for

skyrim - no dlcs

skyrim - all dlcs

skyrim - dawnguard only

skyrim - hearthfire only

skyrim - dragonborn only

skyrim - dawnguard + hearthfire

skyrim - dawnguard + dragonborn

skyrim - hearthfire + dragonborn

 

things would become a lot easier for users:

 

Sure, this is 8 patches instead of 4 but they could have a fomod installer where you can pick the right one. Users would then only have one mod to download and one esp in their load order (instead of up to 4).

Third party stuff would again be loaded strictly after official content and everything would be good.

 

What's the "realest" solution here? I really don't know, considering how uncommunicative the unofficial patch team has been I'd assume the fourth solution is simply wishful thinking and won't happen.

[DoubleYou]

Oddly enough when I tried to do this the unchecked BSA would revert back to checked state when switching from the archive tab and back. It would also do it when double clicking any mod and closing it and with using the refresh button or exiting and restarting MO.

 

Is this normal behavior or do I need to reinstall MO? By the way MO version 1.2.1.

This is an issue with the beta version. It should be solved in the next release: https://issue.tannin.eu/tbg/modorganizer/issues/630

1. Have MO not apply its "magic" on the unofficial patches, thus having them load like the unofficial patch team intended. This is the simplest solution for everyone involved, only drawback is that it breaks the clean layering of mods in MO, at least for these mods

Does an unticked, plugin-loaded BSA take precedence over a ticked BSA, breaking the clean layering of mods in MO? Or will the ticked BSA win the conflict?

[Tannin]

The plugin-loaded BSA always wins.

 

Now that I think about that this won't actually fix the problem completely, right? If the unofficial patches are plugin-loaded and all other bsas are "MO managed" aka "ticked", nothing will be able to override the unofficial patches. Of course this is already true if you unpack all BSAs except for unofficial patches...

[keithinhanoi]

re 4.: My understanding is this, please correct me if I'm wrong:

The load order... <snip> ... is necessary because USKP contains script fixes for the vanilla game that are not compatible with Dawnguard so they need to be replaced by the dawnguard content. The only reason USKP needs to be loaded before DLC stuff is because the DLCs need to be able to overwrite stuff from unofficial patches, correct?

 

 

Yes, that sums it up pretty much. I suppose after reading your rant I could apologize for having knowledge of this (only having finally wrapped my head around it about a month ago) but not alerting you - I was going under the assumption that you were already aware of it. But I was certainly not holding any secret discussions with anyone else. Just the good old keeping it to myself, perhaps due to not having an absolute feeling of 100% confidence about the whole thing.

 

The plugin-loaded BSA always wins.

 

Now that I think about that this won't actually fix the problem completely, right? If the unofficial patches are plugin-loaded and all other bsas are "MO managed" aka "ticked", nothing will be able to override the unofficial patches. Of course this is already true if you unpack all BSAs except for unofficial patches...

Yes, that definitely won't work, because there are other mods which are built around the idea that their scripts and other assets will overwrite ones supplied by the UPs because the UPs are "guaranteed" to load very early. I believe Live Another Life and Cutting Room Floor do this with some UP-supplied scripts for example. But there are plenty of other mods expecting that some of their assets overwrite the UPs "fixed" ones.

 

So - now I think you've reminded me why you suggested not mixing MO-managed and un-ticked BSAs loading based on plugin-order. It's because MO's managed BSA assets + loose files are loaded before the plug-in loaded BSAs.

 

I guess that begs the question of why MO allows BSA management to be changed on a per-file basis rather than a complete on/off toggle switch.

 

Anyhow - a bit of good news:

 

I've just realized that there's another possible solution/workaround to get the official DLCs BSAs and the UP BSAs to load in the correct order in MO:

 

Instead of what I explained in my previous post, keep all the official Beth's .BSAs in /data, and move the UP's .BSAs into /data. 

 

This moves the .BSAs in MO's perspective into the /data folder, as viewed in the Assets tab. 

 

When I did this, they magically got ordered correctly! Not sure how... maybe based on plugin order?

 

However, if that doesn't happen for everybody, then a little-known trick of drag-and-drop re-ordering can be used to get the correct BSA priority load order with the official BSAs followed by the corresponding UP BSAs. This trick only works with BSAs inside a single "group" (with the tiny display toggle triangle next to it.

 

Screenshot time!

 

 

This solution allows for an almost pristine vanilla+DLC /data directory, and won't "break" Steam's auto-update and archive-validation features.

 

And since the virtual /data directory presented to Skyrim combines the contents of the real /Data directory with all active mods in MO - TESV.exe doesn't know that the UP's .ESMs and .BSAs started off in different places.

 

Downside to this option? Every time the UPs get updated, you have to remember to move the new .BSA out of its respective MO mod directory and into /Data.

 

A little cleaner than my previous suggestion though.

 

 

I'm afraid in testing this out I only got as far as loading up a couple of save games, and everything seems to be fine. It's the end of the day in my time zone, and I've got a work project to finish tonight, so I'll leave the testing and ongoing discussion to the rest of you for now.

Edited May 26, 2014 by keithinhanoi

[Tannin]

Yes, that definitely won't work, because there are other mods which are built around the idea that their scripts and other assets will overwrite ones supplied by the UPs because the UPs are "guaranteed" to load very early. I believe Live Another Life and Cutting Room Floor do this with some UP-supplied scripts for example. But there are plenty of other mods expecting that some of their assets overwrite the UPs "fixed" ones.

 So - now I think you've reminded me why you suggested not mixing MO-managed and un-ticked BSAs loading based on plugin-order. It's because MO's managed BSA assets + loose files are loaded before the plug-in loaded BSAs. I guess that begs the question of why MO allows BSA management to be changed on a per-file basis rather than a complete on/off toggle switch.

 

No, sorry, this was misinformation on my part. loose files will still overwrite plugin-loaded bsas.

 

MO allows manual BSA management because there was always the fear that there might be mods that need to be loaded the traditional way.

 

Still, solution 1 I proposed above (and was planning to release) is not feasible: a MO managed BSA can not overwrite a plugin-loaded BSA while loose files do. This means that the moment you have a single BSA not ticked in MO you're in a worsened asset mess of registered bsas vs loose files vs plugin-loaded bsas vs MO managed BSAs.

 

Leaves us with the solutions

2. remove MOs BSA management

3. plugin to import dlcs as MO mods

4. hope for the unofficial patch guys to sort out the mess

5. install unofficial patches manually, outside MO

 

5 is a mess, 4 isn't going to happen, 3 goes against the paradigm that MO leaves your data directory alone.

 

"leaving the data directory alone" isn't only about keeping the game in a vanilla state, I want MO to play nice with other mod managers. But if we remove official dlcs from the data directory that effectively "breaks" the game if you run it without MO and should bethesda decide to update dlcs your imported, now-outdated variants overwrite the updated dlcs.

The DLCs are, as someone else mentioned, effectively mods. But they are steam-managed mods, not MO-managed.

 

This leaves 2 as the only feasible solution, right? Remove the archives tab and all connected functionality and MO works like other mod managers in this regard, get rid of the whole controversy.

 

This of course leaves the BSA extraction vs. Unofficial patch problem untouched but that's really not my fight...

[z929669]

Wrye Bash:

 

TES5Edit:

I just learned that TESEdit method is the only viable one, because Wrye Bash does not support "ONAM subrecords in the TES4 file header"

What I need to know and am not 100% sure on is something I mentioned earlier. Will a ticked BSA be overwritten by an unticked, plugin-loaded BSA in Mod Organizer? That is the one thing I don't know. Could somebody do a simple test? It would be optimal if I'm wrong about that. If a ticked BSA can overwrite an unticked, plugin-loaded BSA, then it is a simple matter of unticking the Unofficial patches and DLC BSAs in the Archives tab, no re-arranging whatsoever.

 

As to my potential problems list, I'm merely listing all the files that are subverted with Mod Organizer's ticked BSA behavior that could potentially be issues, which coincidentally are all the files overwritten by the DLCs from patches loaded earlier. The ramifications outside Mod Organizer are limited only to fixes that potentially could have been overwritten by the DLCs after the switch to false-flagged esp ordering and are most likely non-issues for the Unofficial Patch team, but still worth investigating.

As to the former: Yes, I have confirmed that if a BSA is checked, it will overwrite the plugin-loaded BSA assets (confirmed also by umpacking the BSA). Uncheck and it will not (see my prev post about the details of the facegeom conflicts with the DG vampires).

 

I agree that all issues we know of are associated with the USP false-esm flag change; however, this is theoretically an issue with any mods using BSAs that interact with other mods using BSAs. There may even be other use cases and we just don't know about them ... but this is potentially an issue with any mod interaction and not necessarily and artifact of MO BSA management practices.

 

I have related but slightly different questions. How does MO handle a mod with a BSA but no plugin, or a mod with a BSA and a plugin but the plugin is unchecked in the plugins tab. I expect MO can provide the BSA to Skyrim with the VFS, but what order does this BSA get in Skyrim? If I have two mods each with a plugin and an unextracted BSA, and if mod A has lower installation priority in MO but the associated plugin is higher in mod order, which BSA would have higher priority when loaded into Skyrim?

This was addressed in detail by Tannin and DoubleYou on this thread, and I updated the OP with the consensus info.

 

Finally trying to catch up with this thread. Just want to say off the batm it's very difficult to follow due to all the misunderstandings and inconsistent use of terms.

Agree ... I wish someone on the staff contributing to this thread would help me to keep the OP updated so that all relevant info is consolidated!