Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[DocClox]

I wonder if it's worth submitting a change request/patch to LOOT to handle symlinks correctly?

 

We'd still need a plugin/app to create the links (since Tannin wants MO to run without admin rights) but it might be the cleanest solution. From a STEP perspective, anyway.

[keithinhanoi]

<snip>

I am advocating special treatment of mods within <skyrim>/Data/. If MO can be made to automatically decipher mod names based upon their plugins, that's fine, but it is not necessary. What I really want to see is MO special treatment of the USP content as if it were official content. If this can be accomplished by auto-detection based on file name (or CRC check), fine ... but for the love of Whatever, please allow MO to recognize and specially handle USP content such that it is treated as if it were Bethesda content located within <skyrim>/Data/

<snip>

 

Again, this is not necessary and Tannin has already said no to the idea of an inclusion/exclusion list.

 

It is very possible to read the first 9 bytes of every ESP very quickly to determine whether the ninth byte has the ESM flag. In other words, MO can auto-detect ESPs flagged as ESMs and treat those plugins accordingly. To demonstrate how easy this can be accomplished, in Ruby, for example:

 

<snip>

 

The question then becomes, well, what do you do with them once they've been detected? Tannin doesn't want MO to muck around in the Data directory, so that leaves out moving files. Copying files ruffles the feathers of disk space conservationists. And symbolic links cause LOOT to fail. If MO can't manipulate those plugins in any way, what's left?

 

Actually - Tannin's last post on a proposed short term change to insure the Unofficial Patch BSA assets can load in the correct order along with the official DLC content did not involve treating the Unofficial Patches in a special way.

 

Okay, since Tannin must be busy, let's just do a short review of what he last suggested (with emphasis added by me):

 

I think I'll go with TheBlokes proposal (the original one, see below) for now. It works and is easier to implement than my own, which means I will have a working solution quicker.

My own proposal should still work and I might implement it in a future major update. This gives us more time to consider the implications.

 

I'm still opposed to the idea of treating the unofficial patches special because while it's true that it's unlikely there will be more, there might be further mods from other authors that require similar treatment. Any solution should be as generic as possible, at least in the core program, I have fewer concerns with less genericity in plugins though:

 

What we could do is provide a plugin that checks for "essential mods" and guides the user through the installation, including setup of the right priority. Or fully automates the process.

There is only one problem here: Nexus doesn't want tools to directly download from their service. They are partially financed through ads after all.

To get the most out of such a plugin it would be ideal if we had an alternate download location for the mods.

Also it would be awesome if someone other than me would write the plugin. Not only to save me time but also so we can spread knowledge about the MO plugin interface a bit.

 

One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop.

What do you guys think: Is this a good idea? Unfortunately it would not be possible to discover loose files associated with the foreign-installed esps, nor would I be able to figure out meta information about those mods (i.e. version, nexus id)

So, let's look at TheBlokes original proposal, taken from this post:

 

 

 

Imagine the following:

[*]New user installs MO, on top of completely vanilla Skyrim + DLC installation

[*]When he loads, he sees the Mod list, but it is not empty as now.  Instead, MO has detected that he has a) Skyrim b) Hearthfire c) Dawnguard  (let's say this guy does not have DB), and so it has given him a mod list with the following entries:

[*]Skyrim

[*]Hearthfire

[*]Dawnguard

[*]These mods would be formatted slightly differently to indicate they are vanilla/DLC content, e.g. perhaps they are in bold.

[*]The Skyrim 'mod' would have a hardcoded priority = 0, and no activate/deactivate checkbox

[*]Hearthfire and Dawnguard have normal controls - activate checkbox, and drag/drop priority changing.

[*]Fields such as Nexus ID, rating, Endorse etc would be hardcoded to N/A or similarly disabled.

[*]What has happened behind the scenes:

[*]MO has detected the presence of Skyrim, Hearthfires and Dawnguard.  It has done this probably just with filename matching in the Data directory.

[*]It has created three default, virtual mods, which contain references to their assets.  

[*]These could be entirely 'virtual' references - i.e. no physical filesystem connection at all - or they could be implemented with filesystem soft/hard/junction links, as other posters have detailed.

[*]Either way, MO has some new code that detects certain, named files in the data directory, and has converted these into mod entries.

[*]The user now installs other mods, including the unofficial patches, as normal.

[*]He will be told by STEP guides and other collected wisdom, that in MO he should put each Unofficial Patch mod entry directly after its corresponding vanilla 'mod'.

[*]So once this user has the unofficial patches installed, his mod list with priorities would be:

[*]0 | Skyrim

1 | Unofficial Skyrim Patch

2 | Hearthfire

3 | Unofficial Hearthfire Patch

4 | Dawnguard

5 | Unofficial Dawnguard Patch

6 | .. other mods

So MO would have code to configure itself in the way that many (advanced) users are already doing by hand - treating their vanilla/DLC Data assets as mods, and allowing the DLCs to be re-prioritised.  

 

It would do that without making any physical changes to Data.  It would achieve this either in a purely internal way, or by virtue of automatically creating filesystem soft links from Data to the appropriate Skyrim/DLC mod directory within MO's normal filesystem structure.   So there would now be standard mod folders - "Mod OrganizermodsSkyrim" folder, "Mod OrganizermodsDawnguard", etc - created automatically on startup, and these could contain filesystem links to the appropriate files in Data, or MO could just handle that internally, knowing that the Skyrim mod always contains Skyrim.esm and named Skyrim*.bsa files, and populating the virtual directory and the Conflicts tab as such, without any actual filesystem reference.

 

Then MO would handle its virtual Data, and game loading, exactly as it does now - with no conflicts, because it's treating all assets as mods, with no 'unmanaged' BSAs at all.  Perhaps it would also be necessary to remove the checkboxes against the DLC BSAs in the Archive tab, such that there can't be a situation where the user has ticked, say, the Dawnguard DLC - and so has the ESP - but has for some reason unticked the Dawnguard.bsa file, which would cause that BSA to be loaded regardless because of the esp?  

 

This seems to me to maintain MO's data integrity paradigm.  It does not require removing the existing BSA management feature - which worries me (admittedly more out of principle of not losing potential features, and ignorance of the full ramifications, rather than a firm understanding of any way it would hurt me as a user who extracts 100% of BSAs, including, soon, vanilla/DLC BSAs).

 

In fact I would say it extends one of MO's core paradigms - the ability to have full control and management over a complex mod installation.  It extends it because now all DLCs are manageable just like, and as easily as mods.  If a user wants to start a game without Dawnguard running, he can do so with a single uncheck, in an obvious and logical place; exactly like how he'd disable any mod.  DLCs can now be prioritised however one wants, like mods (though users should be advised that normally, only Unofficial patches should go between vanilla/DLC entries) and can easily have their conflicts/overwrites viewed, just like any other mod.  All within the existing, familiar and effective infrastructure.

 

<snip>

 

 

 

This "core" solution that TheBloke suggested is essentially what I've settled on simulating by making symbolic links of just the official DLC's BSA into the MO mod folders where I've installed the "cleaned" copies of their corresponding .esm plugins.

 

My Unofficial Patches are installed into MO as normal, but because MO recognized the official DLC BSAs being in MO-based mods in the priority list, I immediately gained full control over their priority load order - and the extra added bonus of being able to view all MO virtual /data directory conflict overwrites for the assets of each official DLC and the Unofficial Patches.

 

Let's take a look at what I see in the Conflicts tab in the Information window of Dragonborn with this setup:

 

 

 

I think the advantages here to this fairly simple solution are huge:

[*]Full control over the priority load order of the DLC assets

[*]The ability to set the official DLC and UP BSA assets to load in the same order as their respective plugins, thus matching the native loading mechanism of Skyrim expected by the UPs

[*]Complete reporting of all asset conflict overwrites involving each separate official DLC

On that part of Tannin's suggestion, I can't find any fault whatsoever if he's able to implement it through changes to MO code that is used to generate the virtual /data directory (as TheBloke described.)

 

Then in that same post, Tannin suggested a couple of other things:

[*]An "essential mod" auto-installer built as an MO plugin

[*]Extending the recognition of .esp+.bsa pairs found in the real <skyrim>/data beyond Update.esm/.bsa and the official DLCs, to any third party mods

Of course #2 was actually a suggestion posed as a question, which is what I've been responding to when I have suggested including a warning, regardless of what Tannin ultimately decides to do there. 

 

However, this warning has nothing to do with the Unofficial Patches, because the assumption is they should be getting installed using MO's normal mechanism, and not directly into the real <skyrim>/data directory. If they were, then I think the same warning I've been talking about should apply.

 

Personally, I've settled with the symbolic links of Official DLC BSA put into MO mods as the best workaround for now, and plan on keeping it that way until there's an update to MO which allows the setting or priority load order of the Official DLC's assets.

Edited June 3, 2014 by keithinhanoi

[fireundubh]

Why are your #1s always blank?

 

"Technically it would be easy to discover all esps (plus associated bsas) in the data directory"

 

"Extending the recognition of .esp+.bsa pairs found in the real /data beyond Update.esm/.bsa and the official DLCs, to any third party mods"

 

Not possible without user intervention. ESPs and BSAs aren't inherently associated in any way. Users, or a group of maintainers, would need to manually associate non-core plugins with BSAs. Some of the work could be automated by initially assuming that {filename}.esp is associated with {filename}.bsa, but again, that's not always the case. Nevermind that some mods have multiple BSAs. The user would need to be presented with a list of suggested ESP-BSA associations and choose whether to accept or change them.

Edited June 3, 2014 by fireundubh

[keithinhanoi]

Why are your #1s always blank?

I have no idea - maybe from editing my post? It's driving me a bit crazy, though!It's fixed now, anyhow. Edited June 3, 2014 by keithinhanoi

[Tannin]

Maybe I'm just missing something, but cant you just leave BSAs both unextracted and unchecked in the archives tab? That would make the load order: 1. Official BSAs.2. Third party BSAs loaded via plugin order.3. Loose files. I mean, that's how I've always ran things and I don't recall any issues.

That is one feasible solution, it will treat all BSAs like with other mod managers. If you go this route, make sure you keep all archives unchecked (except the grayed out ones) otherwise strange things might happen (see page 10).What we're discussing here is MOs bsa management that simplifies that load order to:1. loose files and bsasThis is what is a bit broken in connection with the unofficial patches. If you don't use it, you're not affected.

It is very possible to read the first 9 bytes of every ESP very quickly to determine whether the ninth byte has the ESM flag. In other words, MO can auto-detect ESPs flagged as ESMs and treat those plugins accordingly. To demonstrate how easy this can be accomplished, in Ruby, for example:

MO already knows if an esp is a fake esm, but that, to my knowledge, isn't the cause of the issue, a "real" esm could lead to the same issue, no?

To make matters worse, there isn't a way to determine plugin-BSA relationships. How can MO know that AAA.bsa belongs to AAA.esp? File name? Well, there's no Skyrim.bsa for Skyrim.esm.

There is. The game engine does use the file name to represent the plugin-BSA relationship: aaa.es[pm] exists -> load aaa.bsaNot sure if skyrim.esm is special because it's official or if the rule actually is: aaa.es[pm] exists -> load aaa*.bsa but there is a definitive connection based on filename.

File name comparisons is an easy yet unreliable way to validate relationships.

Only if MO determined the relationship different than the engine, but since the engine itself works based on the filename, this is not an issue.

And I'm not even touching on how third-party content in the Data directory doesn't necessarily come neatly packaged in ESP/BSA pairs. Remember the days when people extracted loose files into the Data directory? Well, they still do.

humm, tough cookies, but this is a trade-off:MO never made promises to handle files in data in a predictable way.MO-installed (and managed) BSAs have always overwritten loose files in data, the only difference now is that MO can pull esp+bsa pairs from the "data swamp" so they can be managed properly. This should work for all steam workshop-installed mods, which is a proper bonus imho.NMM installed mods can already be imported through a plugin so I'm not too concerned with mods installed through nmm parallel to MO.This means the only remaining issue is with mods installed outside MO through wrye bash or manually that are loose files but only if the assets are conflicted and if the asset order will only resolve correctly if loose files have the same priority as bsa mods.Honestly I find the use-case that you want to install a mod from steam workshop because it only exists there a lot more reasonable. If someone is using two means of installing mods in parallel he's asking for trouble and shouldn't be too surprised if he gets it.OTOH we could just make it optional and call it a day. ;)

[z929669]

@Keith

I think maybe TheBlokes second post is more comprehensive ... and I do like his theoretical solution if Tannin wants to go with that.

 

I also agree with the notion of user warning 'pop-ups' when a not-so-best-practice situation is discovered.

 

@All

I also like Tannin's thinking about managing assets within /Data/, and I agree with his logic on file name as the key to the plugin-BSA match.

 

Furthermore, I like the idea that asset and plugin prioritizations can be optionally coupled (loose or not).

 

Lastly, Tannin continues to allude a very important point:

MO should play nicely with other mod install methods, and it should preempt user ignorance wherever possible. This is plenty validation for the idea that MO includes methods of dealing with mods within /Data/. I am just going a step further and postulating the situation where the USPs are also installed that way (think of it as an option for those that want to think of the USPs as official mods ... like me ... because I don't agree with any argument that the USPs are doing or fixing anything 'incorrectly', but that is a debate for another thread if anyone wants to have it). Maybe Tannin's ideas about MO recognition of previously-installed mods within /Data/ are also compatible with TheBloke's proposal.

 

@Tannin

Making MO more considerate and accommodating of 'old-school' practices (under Skyrim's revised tenets) and exposing the mechanics to doubters/naysayers and all user classes seems like a worthy endeavor and ultimately why I started this beast of a thread (and no, I am still not ready for the quiz). It seems like you have some good ideas (based partly on others' good ideas), so let us know what you need, if anything.

[fireundubh]

There is. The game engine does use the file name to represent the plugin-BSA relationship: aaa.es[pm] exists -> load aaa.bsaNot sure if skyrim.esm is special because it's official or if the rule actually is: aaa.es[pm] exists -> load aaa*.bsa but there is a definitive connection based on filename.Only if MO determined the relationship different than the engine, but since the engine itself works based on the filename, this is not an issue.

Immersive Weapons.esp seems to work just fine with Immersive Weapons - Meshes.bsa and Immersive Weapons - Textures.bsa.Scripts packed that way, however, seem to fail (e.g., CCO Remade - Misc.bsa), but that may have more to do with the plugin being named Complete Crafting Overhaul_Remade.esp. Perhaps filenames are matched by how they begin rather than exactly? Edited June 4, 2014 by fireundubh

[z929669]

I heard from Arthmoor that Skyrim no longer uses the file name trick to attach BSAs to plugins ... e.g., modA.esp loads modA - 1.bsa AND modA - 2.bsa.  According to him, the file names must match exactly to the left of the extension.

 

This was standard in Oblivion though, so maybe it still applies ... ? It could be tested though.

[SkepticalJoker]

Okay, so the conversation seems to have settled on what seems to be a reasonable solution to me.

 

However, I've just discovered that one solution advocated before is a no go. You should NOT put the Unofficial Patches in the data folder, unless you plan on ditching Mod Organizer. It seems to break everything. I was getting infinite load screens when trying to enter several cells and no amount of conflict resolution would fix it. I was also getting crashes that weren't present before. I discovered this when I tried to make a vanilla profile to test certain mods that I thought might be causing the issue, but no matter what I tried, I couldn't get the game to load through MO. My normal game profile with tons of mods would load (with subsequent ILS/crashes), but a vanilla profile with just the DLCs and Unofficial Patches would not.

 

I decided to remove the Unofficial Patches from the data folder and reinstall them through MO like normal. After that, I was able to load the vanilla profile with no issues. I then went back to my normal profile (now with the Unofficial Patches installed as regular mods) and suddenly the conflict window was showing a ton of mods that needed to be reordered. I then loaded up my save and tried to enter the nearest cell that was giving me a consistent infinite load screen and the issue was resolved; it loaded up just fine. It seems the problem is that since MO can't account for conflicts with mods installed in the data folder, you can end up with a complete mess in your priority sorting unless you completely ditch MO. Meaning no mixing and matching mods through MO and mods in the data folder.

 

But it also seems to create an even bigger issue, since I can't explain why the vanilla profile wouldn't load, but my normal game profile would (albeit with major problems). Why would that be? Even when I tried to deactivate the Unofficial Paches in order to load just vanilla Skyrim, it wouldn't work.

 

tl;dr - It should be strongly advised to keep all mods out of the data folder when using MO, no matter what. The only exceptions I guess would be the official DLCs, although I'm strongly considering just moving those into MO as mods.

Edited June 8, 2014 by SkepticalJoker

[Rubber_Soul]

Okay, so please excuse me if the question I'm about to ask has already been answered (pretty sure it has), but after untold hours of reading, this question still lingers.

Up to this point I have already extracted all the bsas using MO except for the Unofficial Patches. I don't know whether this is STEP's fault for making it seem as though there's some advantage to bsa extraction, or my own for not making absolutely sure that, as a standard user, I fully understood the ramifications for doing so (and until this thread, I thought there were none).

So after completing my STEP installation, the only bsas I have in my archives tab are the Unofficial Patches, and the DLC, and they are all unchecked, while their plugins are checked, with everything else mostly identical to the extended STEP setup, and this is how I've been playing my modded Skyrim up till finding this thread.

So my question basically is, is there anything particularly wrong with my setup? Because from what I gather from this thread, I think there might be.

A more specific question would be: Would the mods that are supposed to be overwritten by the Unofficial Patches (like "Distant Decal Fix", and "Consistent Older People") be so? I ask this question because of this part in the OP:

 

 

• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the  :!:warning).

The part in red is what has me worried.

Anyway, sorry for being long-winded (can't help it), and thanks in advance to anyone who responds.

Edited June 8, 2014 by Rubber_Soul

[DoubleYou]

Distant Decal Fix is the only STEP mod that is to be overwritten by the Unofficial Patches. Although apparently 2.2.8 has USKP overwriting Consistent Older People, that must have been an oversight, as that mod is made explicitly compatible with USKP and uses a BSA that obviously must be intended to load after the USKP. Of course, the files in conflict are unlikely to cause any real trouble (a mesh and a texture).

 

The fact that the only BSAs you have are the Unofficial Patches and these are unchecked means that your configuration is safe. It is mixing checked and unchecked BSAs for you that will cause trouble. The only issue that will be present is Distant Decal Fix will be fixing everything and not using the USKP fixes, which really should not be a problem.

[Rubber_Soul]

Thank you! That is exactly how I thought it would work.

I'm just glad the STEP guide now explicitly states that there are no benefits to using loose files over BSAs in MO (pretty sure it encouraged unpacking some time ago), so that other "average" users wouldn't make the same mistake. I'd say there even needs to be a sort of warning against it, unless the user knows what they're doing.

[Octopuss]

I am a bit (a lot) confused by all that's been happening lately, and would like something cleared:

Does MO treat loose files and BSAs equally or not? I've always lived under the impression it made no difference (and I think Tannin even said so), but now that I read the first post and with the recent discussions, I hear the loose files still take priority. I don't get it at all :(

[Aiyen]

I know I am going to stir the hornets nest again.. but overall the tone of the OP is a bit too much "Uhhhhh BSA extraction... be afraid, be very afraid".

Which I am not really a fan of. 

 

I am also confused why there is still a performance related "pros" at all... The difference between extraction and non extraction is so minimal that calling it a performance increase is kinda misleading. Ofc it is system dependent, but for all computers able to even run skyrim in the first place it should not be an issue at all. 

The reason between doing one or the other should never be performance... it is all a matter of how easy access you want to your game assets.  

[DoubleYou]

It is because MO treats BSAs as loose files that we have had this discussion, and why data BSA mods are now added to MO, as proper conflict resolution requires this in some cases.