Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[TechAngel85]

@TechAngel85: I don't see why install order matters at all. You know you can drag mods in the middle pane to reorder them, right?

Middle pane? I have a left and right pane...no middle. :O_o: Btw, I'm using installation and mod order interchangeably here.

[DoubleYou]

Dunno if we helped Unofficial Dragonborn Patch fix a bug or not, but the changelog for 2.0.4 now mentions dunprogressivecombatscript, which means that is no longer a problem if you use Unofficial Dragonborn Patch.

[fireundubh]

Middle pane? I have a left and right pane...no middle. :O_o: Btw, I'm using installation and mod order interchangeably here.

If you click the Filter button at the bottom left, you can open the real left pane.

 

Also, remove installation order from your vocabulary with regard to MO because the order in which you *install* mods is rendered moot by MO.

Edited May 29, 2014 by fireundubh

[keithinhanoi]

Second, several of the Team is currently testing STEP in a non-extracted BSA setup. One of the main reasons, we recommended BSS extraction came from the WB days. It is more necessary when using WB. With MO, this is not the case so that recommendation will likely be removed. The ONLY time we're going to recommend BSA extraction will most likely be for mods that need assets hidden so that a lower priority asset can win the conflict.

I think changing to this recommendation is a very good idea. Tannin has already mentioned having a significantly large number loose files is more resource intensive, and potentially leads to a significant increase in drive space used (a consideration for SSD users if the firmware doesn't already automatically compress written data.)

 

Also, MO already has a post-installation BSA extraction feature - right click on the BSA in question in the Assets tab of the right-hand pane to do this - which means you don't necessarily have to unpack every BSA at mod installation / update just because you think it gives you more control over the assets. You have that control at any time - so I think it's better to do that when you realize there's a conflict situation that requires unpacking the BSA. 

 

In addition - unpacking BSAs for mod updates has a potential "catch-22" (see my explanation under TheBloke's quote below.)

 

Tech - I'm sure you are well aware, but I think it still needs to be pointed out that just like a .esp plugin patch can carry foward records from mods earlier in the load order priority, required conflicting BSA assets from lower priority mods can be extracted and moved into a special "assets patch" mod that can be set with a higher priority than mods with BSA containing the unwanted conflicting assets.

 

I don't know how many cases there are in the current STEP lineup which would require BSA extraction, but this alternative approach is something to keep in mind. The bottom line here is that MO allows for this kind of flexibility in a solution for conflict management.

 

Many seem to be against this whole idea of tying the two together though. I suspect there will be a lot of "before running LOOT select all mods, toggle auto-sorting off, now run LOOT, now move mod XYZ to this location, etc etc".

 

That why MO will function mainly as it is functioning now. As for STEP, we'd have to come up with some workarounds on the Guide due to all the mods that would be moved around by the auto-sort, but it could work.

I don't really see the problem here. In Tannin's suggested change - he specifically says that sorting of mod priority/"install" order based on plugin-load order could be disabled to allow the user to manually set mod priority - just like we have now.

 

But instead of the mess that is currently caused by mixing "MO managed" and plugin-load order managed BSA assets, MO's "dark magic" method to build the Virtual File System /Data directory would be used for both priority sorting methods ("auto" and manual) - which the current version of MO doesn't do now (currently it releases plugin-load order managed BSAs to be handled by Skyrim's plugin+BSA loading mechanism.)

 

As a result of the proposed change, this would allow STEP to recommend turning off mod priority auto-sorting (as you've called it) for all mods except the UPs, and then start ordering them according to the STEP Guide - pretty much as you have it now.

 

So there are actually no workarounds needed for what Tannin's proposing. I think the only thing I would suggest here is the ability to globally toggle mod priority auto-sorting & manual sorting for all mods simultaneously, and more importantly, a preference setting to change the default from mod priority auto-sorting to manual. From the STEP Guide perspective, this would simplify things so the steps to add would be:

[*]Before installing any mods, open the Settings menu, click the Advanced tick-box to enable it, and the un-tick the option "By default, sort priority of mods with BSA assets according to plugin load order", so by default the priority of all mods installed can be manually set.

 

[*]After installing the Unofficial Patches, in the left-pane Mod Priority list, enable the tick-box in the Mod Priority Auto-Sort column for all four Unofficial Patches.

And that would be it. So the UPs' BSA load order would be set by plugin-load order as set by LOOT, meaning they would be in the correct load order slots (just after the respective DLCs,) and the priority order for all other mods would be manually set. 

 

The importance here, as Tannin stated is for non-STEP users (and there are a few out there last time I checked,) the default would be that all the priority/"install" order of all mods with BSAs would be set according to accompanying plugin load order.

 

The SMIM example that you've mentioned wouldn't actually apply here, because in Tannin's new "compromise" version of his suggestion, all mods with loose files would have manual mod priority sorting by default - and SMIM has no BSA, so we don't have to worry about any of its .esp plugins setting its place in the mod priority order. This is why in my invented example for the global mod priority auto-sort setting, I called it "By derfault, sort priority of mods with BSA assets according to plugin load order." You turn that off, and suddenly MO would work just the same as it does now - except you can still toggle auto-sorting on a per-mod basis without the "mess" as Tannin described.

 

But I've found that the fix for this actually causes a new, more serious issue:  whenever an update patch is applied (example: upgrading Interesting NPCs from 3.05 to 3.06 using the 3.05->3.06 incremental patch archive), and that patch contains a BSA, any files in the patch BSA will not be extracted if they already existed in the original mod.  

 

So patch updates will be broken for any mods that use incremental patches, where the patch contains a BSA file that is meant to update files in the original release, and where the user has chosen to Extract all BSAs.  The "don't extract from BSA when existing file is found" rule is applying always, and that is not desirable in the case where the user is choosing to Merge an update into an existing mod, and is extracting all BSAs.

This unfortunately creates a major "catch-22" problem when installing updates of BSA + loose file mods which are meant to be installed using the merge option instead of replace (because the update doesn't include the full set of assets.)

 

So, say the author has a "main" BSA with the default assets, and their FOMOD installer includes options with loose files that are intended to override some contained in the BSA. It's normal of the author to expect this to work because without any mod manager or with NMM, the loose files load after all BSA. So I wouldn't blame the author for doing anything wrong by setting things up this way.

 

Now, the latest update of MO deals with this correctly if you want to unpack/extract the mod's BSA to loose files. So, when installing, essentially MO unpacks the BSA first, and then adds the loose files that were installed, overwriting anything unpacked from the BSA with the same name. So that works just as expected.

 

However, if you install an update for the mod, which just includes a replacement BSA, you're going to run into trouble.

 

Because we've previously unpacked the mod's BSA, now everything is mixed together as loose files. How does MO know which ones were originally supplied by the installer as loose files? It doesn't. So when you install this update using the merge option, and allow MO to unpack/extract the BSA, it's actually going to overwrite some of the files that were originally supplied by the previous install as optional loose files.

 

Thinking through this, I can't think of any way of MO knowing what to do automatically, without somehow keeping track of which files were previously supplied as loose files and not in the unpacked BSA.

 

However, there is a bit of a workaround you could use in this case, and it means you need to keep a backup of the previous install archive, or download it again. So when a mod has an update which just contains a replacement for a main BSA that you had previously unpacked, and not any of the optional loose files, you do this:

[*]Re-install the previous version of the mod again, selecting overwrite, but this time don't unpack the BSA.

[*]Install the update, selecting merge, but also don't unpack the BSA.

[*]Go to the Assets tab in the right-hand pane, and find the BSA for the mod in question

[*]Right-click on the BSA, and select Extract...

VERY IMPORTANT: This workaround is based on the assumption that Tannin also applied the unpack-BSAs-not-to-overwrite-loose-files fix in MO 1.1.2 to this other method of unpacking a BSA.

 

Tannin - Did you also apply that fix to the Assets tab Extract BSA contextual menu feature?

 

 

If you click the Filter button at the bottom left, you can open the real left pane.

 

Also, remove installation order from your vocabulary with regard to MO because the order in which you *install* mods is rendered moot by MO.

Not sure what Tannin regards it as, but I've always thought of the Filter list as a "fly-out", not a real member of the "pane society" who can never be made to disappear from view! 

 

Personally, I don't look at the mod priority list as install order either, but started referring to that list/pane as the priority/"install" order list in this thread because other people / STEP refers to it as the "install" order list, and wanted to make sure people were able to following what I'm talking about.

Edited May 29, 2014 by keithinhanoi

[TechAngel85]

If you click the Filter button at the bottom left, you can open the real left pane. Also, remove installation order from your vocabulary with regard to MO because the order in which you *install* mods is rendered moot by MO.

You haven't been around that long but "installation order" is a term frequently used around here from the STEP Guide. I understand what you are saying; however, anyone that follows the STEP Guide in order will have the proper "mod order" set up automatically (meaning there is no need to move mods around unless you accidentally skipped a mod or something) because they "installed" the mods in the order they are on the Guide; thus, having the proper "mod order" already set up. I just did this today for 2.2.9 sooo...yeah.  

snip/ I don't know how many cases there are in the current STEP lineup which would require BSA extraction, but this alternative approach is something to keep in mind. The bottom line here is that MO allows for this kind of flexibility in a solution for conflict management.

I just finished installing 2.2.9 today and there are none as of yet. All files we instruct to be hidden are either already in loose file format or are available to download in this format. 

I don't really see the problem here. In Tannin's suggested change - he specifically says that sorting of mod priority/"install" order based on plugin-load order could be disabled to allow the user to manually set mod priority - just like we have now. snip/ As a result of the proposed change, this would allow STEP to recommend turning off mod priority auto-sorting (as you've called it) for all mods except the UPs, and then start ordering them according to the STEP Guide - pretty much as you have it now.

I don't think that would be ideal. We'd most likely want the sorting on the mods that have PMOP errors which is one of the main reasons of doing the auto-sorting in the first place. Disabling the sorting on all mods but the UPs would be counter intuitive to this. 

snip/ The SMIM example that you've mentioned wouldn't actually apply here, because in Tannin's new "compromise" version of his suggestion, all mods with loose files would have manual mod priority sorting by default - and SMIM has no BSA, so we don't have to worry about any of its .esp plugins setting its place in the mod priority order. /snip

It was my understanding that ALL mods with plugins would be auto-sorted by default, not just the ones with a BSA+ESP combination. Perhaps is read or understood incorrectly.

[keithinhanoi]

It was my understanding that ALL mods with plugins would be auto-sorted by default, not just the ones with a BSA+ESP combination. Perhaps is read or understood incorrectly.

Tanin explained a variation of his suggestion in this post, exactly with these two sentences: 

Say we did a compromise of my last suggestion: MO by default activates auto-sorting for mods containing bsas but not for loose-file mods (You could of course always toggle it)[/size] In this case, by default asset ordering in MO would work almost exactly like in NMM (except that BSAs would still be able to override loose files if their priority is higher)

So, if MO was implemented this way, then all mods without BSAs would have manual asset priority control by default. However, it would still allow you to place a mod with loose file assets between two mods that are priority auto-sorted based on plugin order. Now - that just leaves the problem of being able to present the STEP Guide in "tidy" categorized sections, right? This is of course in direct conflict with Tannin's suggestion that the mod priority order for auto-sorted mods would match plugin load order, and you've said that STEP would probably want to use the priority auto-sorting method for mods with BSAs: 

I don't think that would be ideal. We'd most likely want the sorting on the mods that have PMOP errors which is one of the main reasons of doing the auto-sorting in the first place. Disabling the sorting on all mods but the UPs would be counter intuitive to this.

So, if all mods with BSAs have their priority ordered according to the plugin order as set by LOOT, then then it breaks the whole concept of having the STEP mods listed in sections.  The key thing here is that STEP has been setting MO mod priority via the guide's list, and except for the conflict section that's actually quite arbitrary when compared to the load order that BOSS / LOOT would set them to. (Actually the BOSS Masterlist is in many ways arbitrary, but that's a discussion for another thread!) I think we're forgetting that there are multiple sortable columns in the MO Mod priority list (left- or middle- pane, whatever).  If you want to keep STEP's mod list into tidy sections, then consider having the user set up custom STEP Categories in MO, and then setting that for mods as they're installed so that they can be listed in tidy sections, if the list is sorted by category. However, the bottom line is that with Tannin's suggested changes to MO, you would still want to produce a new list showing the correct (numerical) mod priority. Anyhow, there's really no way of getting around this if STEP would use the new suggested auto-sorting feature. You simply can't list the priority by section, alphabetically AND also set the priority of mods with BSA based on plugin load order.

Edited May 29, 2014 by keithinhanoi

[TechAngel85]

I'm not too concerned right now about the categories as I am the mod order. The categories are a way that we can keep things "tidy" but still resolved conflicts in the proper manner. The mods are in a specific order on the Guide for conflict resolution purposes. Everything below the Conflicting Graphics section is meant to overwrite assets from this section. Where you find a mod not in alphabetical order (AOS, Non-Essential Children, etc), those mods are place there to overwrite assets from mods that are above them on the Guide. The whole Guide would have to be arranged to compensate for the auto-sort. The only way to know how that will work is by doing it I suppose, so I'll work on a new profile that is arranged according to plugin order, where BSA's are involved. The loose assets I'll leave in their proper order and we'll see how this goes.

 

Thanks for clearing some of that up for me.

[namtrahj]

I've been following this thread to the best of my ability, and to be honest a fair amount of it is over my head. My biggest source of confusion, I think, is keeping straight hypothetical future release vs. beta update vs. last official release.

 

In my case, I am using the last official release of MO--I have NOT updated to the new beta. I had always extracted all BSAs, but I am rethinking that due to this thread. I've added the three DLCs as mods and left the BSAs intact. I'd also like to realize the performance gain of keeping BSAs unextracted as mentioned by Tannin and others.

 

So, given my version of MO, will I have any priority issues if I reinstall/update some mods and do not extract the BSAs, as long as I make sure that ALL BSAs are checked/ticked in the Archive tab? That is the way to ensure that all mods, whether loose files or BSAs, are treated equally by MO, correct? Problems start when a BSA is NOT checked in the Archive tab, right?

 

Also, again keeping in mind that I am not using the beta, can I update 3DNPC without issue? My existing install is loose/extracted, so can I install the update by merging? And should I extract the update BSA since the existing install is extracted?

 

I apologize for the questions and I hope I'm not fundamentally misunderstanding anything. I'm just a bit lost on which release of MO is being discussed sometimes and I've lost sight of what I should do with my existing install. Hopefully clarifying/reiterating some of this will help some other people that might be following along and realizing that they are becoming increasingly lost. I appreciate the help, guys.

[DocClox]

The mods are in a specific order on the Guide for conflict resolution purposes. Everything below the Conflicting Graphics section is meant to overwrite assets from this section.

It occurs to me that a STEP metatdata file for LOOT might be very useful here. No need to worry about ordering, just sort with Loot.The problem would be that any update to Loot's "masterlist" (if it's still called that) might overwrite the STEP ordering.I wonder if Loot would be willing to consider multiple cascading metadata files...

[TechAngel85]

@NamtrahjI believe you are correct in your assumptions. As for 3DNPC, the best solution is to simply create a separate mod directly under it for the updates.

 

@DocClox

We're planning on submitting a few reports to the LOOT team. @OthersI just finished reordering the mod list to reflect a BSA+ESP plugin priority for STEP 2.2.9. It would seem that it's not that big of an issue. Yes, mods are not within their sections but at least 50% of them had no conflicts so it didn't matter where they were placed. The other 50% seem to have kept the conflict priority set up by the Guide. I did have to move Distant Decal Fix and Consistent Older People above the USkP to maintain those conflict resolutions. The only one I'm not 100% certain about are Follower Safety and Traps Make Noise. The have be switched so the conflicts have been switched as well (I think it's a script conflict; however, I followed MO's PMOP in this suggestion). Speaking of PMOP, reordering the BSAs in this manner also resolved all PMOP issues but one, "Move aMidianBorn Book of Silence after Non-Essential Children - Hearthfire". This is fairly impossible to do and keep the conflict resolution of the STEP Guide.So, even though the mods aren't maintained in their "tidy" order, I see no real issues with Tannin's compromised solution if it works out like I have the list ordered.

[Tannin]

I think I'll go with TheBlokes proposal (the original one, see below) for now. It works and is easier to implement than my own, which means I will have a working solution quicker.

My own proposal should still work and I might implement it in a future major update. This gives us more time to consider the implications.

 

I'm still opposed to the idea of treating the unofficial patches special because while it's true that it's unlikely there will be more, there might be further mods from other authors that require similar treatment. Any solution should be as generic as possible, at least in the core program, I have fewer concerns with less genericity in plugins though:

 

What we could do is provide a plugin that checks for "essential mods" and guides the user through the installation, including setup of the right priority. Or fully automates the process.

There is only one problem here: Nexus doesn't want tools to directly download from their service. They are partially financed through ads after all.

To get the most out of such a plugin it would be ideal if we had an alternate download location for the mods.

Also it would be awesome if someone other than me would write the plugin. Not only to save me time but also so we can spread knowledge about the MO plugin interface a bit.

 

 

One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop.

What do you guys think: Is this a good idea? Unfortunately it would not be possible to discover loose files associated with the foreign-installed esps, nor would I be able to figure out meta information about those mods (i.e. version, nexus id)

[EssArrBee]

The UPPs are on many sites and it wouldn't be all that hard to use one of them. Arthmoor's site, iguadons or something, Dark Creations, and a bunch others host them. Hell, we might even be able to get them to host off our site somehow.

[Kuldebar]

Also, remove installation order from your vocabulary with regard to MO because the order in which you *install* mods is rendered moot by MO.

While this is awesome, it will make it very hard for some people (me?) to get their heads around without a blatant WYSIWYG type presentation.

[noobzor]

I just want to testify here real quick.  After reading through this thread the other day, I decided to try undoing all of the BSA unpacking that I had done just to see what noticeable differences there were.  It ended up being somewhere around 150 mods that I had unpacked.  After I was finished reinstalling and leaving in the original BSA, I was pleasantly surprised to discover significant performance improvement in my game...much more than I expected.  Loading times are a mere fraction of what they were with loose files, and all of the stuttering that I had when loading cells has completely gone away.  Reverting to the bsa's was the only change I made to the game.

 

A few days ago, I was sure that unpacking BSA's was the way to go, but I have seen the light.  From now on, no more unpacking BSA's without a good reason.  

Edited May 29, 2014 by noobzor

[EssArrBee]

@noobzor, thanks for confirming this, but I would also point everyone to a nice quote from Ethatron:

A little calculation

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

• Harddisk-throughput: 20MB/s
• Memory-throughput: 2000MB/s
• zlib-throughput: 80MB/s
• uncompressed size: 100MB
• compressed size: 60MB

This yields:

• Reading 100MB at 20MB/s from disk takes 5s.
• Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.

In case the data is very small it's probable that seeking time will start dominating the times:

• Reading 1MB at 20MB/s from disk takes 0,05s.
• Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.

Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

 

 

I believe a couple SSDs are now passing the 500MB/s second mark for SATA 3 and SATA Express is also starting to show up in a few enthusiast/professional products, but that still will not make up the time gap until SSDs are in the +1.2GB/s range. Also, decompression speeds will increase with new hardware and more efficient algorithms, so the performance argument is pretty dead.