Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[Tannin]

I know what lzma is and I know that MO can extract them, I wrote the code. ;)

But on BSAs only zlib is supported. Please post the name of any one bsa that is supposed to be lzma compressed. I'm fairly certain binwalk is wrong.

Please also note that while BSAs use zlib compression they are not zip files! They don't have any signature other than BSA.

[fireundubh]

Please post the name of any one bsa that is supposed to be lzma compressed. I'm fairly certain binwalk is wrong.

Already did.

 

 

Signature Analysis Results

 

If we could adapt this BSA header information to a magic signature, Binwalk might produce different results, but I'm not sure how useful that would be in the long run. The object of doing all this was simply to determine the type of compression used by Skyrim.

Edited May 31, 2014 by fireundubh

[Tannin]

zlib: https://www.uesp.net/wiki/Tes4Mod:BSA_File_Format#Compressed_File_block

This has been well known for ages, how else do you think people are able to extract them? The whole file format is known and documented, same goes for almost all other file types used by that engine.

 

Both update.bsa and Skyrim - Misc.bsa are zlib compressed, there is no lzma data or header there. Binwalk is wrong.

[fireundubh]

zlib: https://www.uesp.net/wiki/Tes4Mod:BSA_File_Format#Compressed_File_blockThis has been well known for ages, how else do you think people are able to extract them?

Magic.I linked the same page. That page indicates that there is uncertainty about various elements of the BSA format.And, FYI, I was just trying to answer to EssArrBee.Also, "Skyrim - Misc.bsa" may not be compressed at all. Binwalk finds zlib headers for other BSAs.edit: Binwalk reports many lzma false positives because lzma has no magic bytes for which to create a reliable signature. Edited June 1, 2014 by fireundubh

[keithinhanoi]

Let me put it this way: If MO imports Dawnguard, say, and leaves a mod that modifies Dawnguard with a BSA in the real data directory, the mod's assets will be overwritten by Dawnguard. Therefore, Tannin not only should import all mods from data, but also must in order to maintain compatibility in some cases. Certainly, in a perfect world where all users leave their data directory clean, this wouldn't be necessary, but Skyrim isn't Utopia.

Understood & agreed, but that only addresses mods with all assets contained in a BSA. As soon as were talking about third party mods in the real /Data directory which provide loose file assets, then it becomes a mess. MO won't be able to discern which loose files came with which plugin, and so how would the user then have control over the load-priority order of those loose files? Should MO make a special "fake" mod with all of the non-Vanilla/DLC loose files? Can you see how this gets too complicated pretty quickly?This is why I think a stern warning message is needed advising the user to move those mods out of the real /Data and into their own mod folders in MO. Unfortunately, in absence of the original installers for those mods found in the real /Data, it would be difficult for MO to automate the process of moving them into the MO mod priority/"install" list due to the loose files issues I mention above.Either way, I think there should be a clear warning message about this, because when a user who has left third party mods in their real /Data directory complains that mod X isn't working for them, they can be reminded of that warning. Edited June 1, 2014 by keithinhanoi

[keithinhanoi]

Also, "Skyrim - Misc.bsa" may not be compressed at all. Binwalk finds zlib headers for other BSAs.

It's not compressed at all.

• Skyrim - Misc.bsa size = 177,490,631 bytes
• Unpacked size of files = 176,843,788 bytes
• Re-packed w/BSAOpt, compression OFF = 167,956,480 bytes

Note that the re-packed BSA is smaller because BSAOpt excluded 453 duplicates, in which case the packed file is referenced when the game looks for its duplicate's file name.

 

(As a side note, I still don't understand how BSAOpt decides two files with different names are duplicates - does it use a CRC check?)

 

Anyhow, the slightly smaller size of the unpacked files compared to the original Skyrim - Misc.bsa can probably be explained as the size of the file header, I imagine.

 

Skyrim - Misc.bsa does include .gid files - maybe these are lzma compressed, and that's why you're getting the signature?

Edited June 1, 2014 by keithinhanoi

[DoubleYou]

Understood & agreed, but that only addresses mods with all assets contained in a BSA. As soon as were talking about third party mods in the real /Data directory which provide loose file assets, then it becomes a mess. MO won't be able to discern which loose files came with which plugin, and so how would the user then have control over the load-priority order of those loose files? Should MO make a special "fake" mod with all of the non-Vanilla/DLC loose files? Can you see how this gets too complicated pretty quickly?This is why I think a stern warning message is needed advising the user to move those mods out of the real /Data and into their own mod folders in MO. Unfortunately, in absence of the original installers for those mods found in the real /Data, it would be difficult for MO to automate the process of moving them into the MO mod priority/"install" list due to the loose files issues I mention above.Either way, I think there should be a clear warning message about this, because when a user who has left third party mods in their real /Data directory complains that mod X isn't working for them, they can be reminded of that warning.

No, I'm pretty sure a stray loose files "mod" would not be needed, as Mod Organizer doesn't include conflicts from the data directory, and since they were loose, they would overwrite the BSAs. Mod Organizer makes BSAs able to overwrite loose files when checked by scanning for loose file conflicts in installed mods and removing them from the hooking into Skyrim real data directory. Since those loose files are already in data, Mod Organizer will leave them there, and there should be no problem at all. If, of course, some other mod in Mod Organizer has a loose file that conflicts with the one in data, the mod will win the conflict.Nevertheless, mods in actual data directory is not good, and I would agree with you that a strong warning should be issued if this is found, but that is not a valid argument against importing all plugins as mods inside Mod Organizer, which indeed would be a good feature.

[fireundubh]

Skyrim - Misc.bsa does include .gid files - maybe these are lzma compressed, and that's why you're getting the signature?

I don't know. I moved all of the nonessential signatures to separate plugins, including the wide variety of possible lzma signatures.I also created a magic signature for BSAs.

# BSA archives0       string          BSAx00                 BSA archive>4      byte            0x67                    b, version: %d>4      byte            0x68                    b, version: %d>8      byte            0x24                    b, folder records offset: %d

When I run Binwalk on Update.bsa, I now get:

DECIMAL   	HEX       	DESCRIPTION-------------------------------------------------------------------------------------------------------------------0         	0x0       	BSA archive, version: 104, folder records offset: 367422937   	0x7143D9  	Zlib header, best compression, uncompressed size >= 655367688740   	0x755224  	Zlib header, best compression, uncompressed size >= 461827704627   	0x759033  	Zlib header, best compression, uncompressed size >= 1020487737852   	0x7611FC  	Zlib header, best compression, uncompressed size >= 655367855149   	0x77DC2D  	Zlib header, default compression, uncompressed size >= 812617880806   	0x784066  	Zlib header, default compression, uncompressed size >= 1068517913871   	0x78C18F  	Zlib header, default compression, uncompressed size >= 869927942406   	0x793106  	Zlib header, default compression, uncompressed size >= 9830416855268  	0x10130E4 	Zlib header, default compression, uncompressed size >= 9830416902615  	0x101E9D7 	Zlib header, default compression, uncompressed size >= 1757916908715  	0x10201AB 	Zlib header, default compression, uncompressed size >= 9830416955011  	0x102B683 	Zlib header, default compression, uncompressed size >= 6130616978305  	0x1031181 	Zlib header, default compression, uncompressed size >= 6553617038589  	0x103FCFD 	Zlib header, default compression, uncompressed size >= 6553617109606  	0x1051266 	Zlib header, default compression, uncompressed size >= 9830417156267  	0x105C8AB 	Zlib header, default compression, uncompressed size >= 7233817179168  	0x1062220 	Zlib header, default compression, uncompressed size >= 9830417213805  	0x106A96D 	Zlib header, default compression, uncompressed size >= 9830417261891  	0x1076543 	Zlib header, default compression, uncompressed size >= 13107217306910  	0x108151E 	Zlib header, default compression, uncompressed size >= 9830417353626  	0x108CB9A 	Zlib header, default compression, uncompressed size >= 65536

On Skyrim - Misc.bsa:

DECIMAL   	HEX       	DESCRIPTION-------------------------------------------------------------------------------------------------------------------0         	0x0       	BSA archive, version: 104, folder records offset: 36

I think that makes clear that the latter file is not compressed, so I can effectively use Binwalk to determine whether BSAs are compressed without extracting them.Probably not that useful, but still (nerdy) cool! Edited June 1, 2014 by fireundubh

[z929669]

Actually, if a user has installed ESP+BSA paired mods in the "real" /Data directory, then MO will place their BSAs at the top (earlier) in the BSA priority list. If MO doesn't pull them in as "special" mods that show up in the left-hand (middle) mod priority/"install" list pane, then it's still possible to subvert the BSA load order as compared to plugin load order.

 

If MO were changed so those mods were brought in as "special", allowing priority/"install" placement to set load order of BSA assets (but not loose files - which would be impossible,) then the user could move those mods in the priority/"install" list so that it matches load order, if they wish.

 

Possible issues resolved by doing this are vanilla-origin scripts/meshes/textures that the mod expects to overwrite by virtue of the game's normal BSA-tied-to-plugin load order. It also becomes even more important when thinking of the UPs, because if a user has any mod in the real /Data directory now which has BSA assets of the same names as anything supplied by the UPs, the UPs version will overwrite the ones of the other mod.

 

No - I think one of two things would be a very good idea here. Either:

[*]Tannin's suggestion of bringing all plugins (and their BSAs, if they have them) into the MO priority/"install" list as "special" mods, or

[*]a warning message that third party plugin-based mods were found directly installed in the real /Data directory, with a summary of the potential issues caused by that, but NOT bring those mods into MO.

Personally, I like #2 more, because really the whole point of MO is not to install mods directly into the real /Data directory, and if any of those mods included loose files assets (not allowed by Steam Workshop, right?) then it leads to further potential problems (and general confusion.)

 

Not everybody has a machine with a fast processor, RAM, bus, SATA III, an SSD with on-the-fly firmware-based write compression.

 

So, please if any tests are done, include test systems with lower-end specs.

 

Though I don't know for sure, it should also be noted that with the STEP lineup there's likely a mix of BSAs with compression and no compression - no compression being a choice by mod authors because BSA compression doesn't seem to work with audio (and other?) file assets.

 

I agree with Keith on all points, but I actually prefer option #1

 

Also, we need to get off the whole SSD/compression debate unless it directly pertains to the topic at hand. It is a sidebar that is detracting from the real issue.

 

Since it has not been addressed:

... snip/

One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop.

What do you guys think: Is this a good idea? Unfortunately it would not be possible to discover loose files associated with the foreign-installed esps, nor would I be able to figure out meta information about those mods (i.e. version, nexus id)

I very much like this idea. It can only help.

 

I also don't see any major reason why you could not eventually implement your original idea (all mods, loose and BSA sort with plugins), especially if we have the option to turn off this feature at the mod level (and globally too would be nice). Action only on mods with BSAs does not make sense, since any mod could (and maybe should) bepacked within a BSA. What is and is not affected would then be arbitrary and nonsensical. I favor the original idea of prioritizing ALL mod assets with plugin prioritization as a toggleable feature. This is a feature ADD, not a detraction.

 

Also, see my previous post about differential text/background colors of mods managed this way versus not in both mod and plugin lists. This way users can tell what list prioritization is relevant to mods in the corresponding lists.

 

I also agree with DoubleYou.

[Kelmych]

 

(As a side note, I still don't understand how BSAOpt decides two files with different names are duplicates - does it use a CRC check?) 

Based on the log files it seems to use a CRC check.

[phazer11]

Based on the log files it seems to use a CRC check.

I do too, based on this post; granted it's probably LOOT doing CRC checks but... I do find that a little odd.

[keithinhanoi]

Since it has not been addressed:

... snip/One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop.What do you guys think: Is this a good idea? Unfortunately it would not be possible to discover loose files associated with the foreign-installed esps, nor would I be able to figure out meta information about those mods (i.e. version, nexus id)

I very much like this idea.

 

z - I'll say it one last time: either way there should be a warning and a strong recommendation that the user uninstall all third party mods from the real <skyrim>/Data directory and install them using MO's mechanism.The main reason for this is because of what I explained about how loose files in /data would be handled.Currently, as far as I understand it, if MO is "managing" all BSA assets (in other words, all BSAs in the Assets tab are ticked/checked and don't have the little warning symbol), then the loose files in /Data are loaded with priority 0. The means that regardless of where in the mod priority/"install" order list that you place any third-party mods that MO would find, any loose files installed by those mods could potentially be overwritten by any other mod installed using MO normal mechanism.I imagine a bit of a scary situation in which a new MO user who has been using NMM sees all of their mods in <skyrim>/Data listed in MO (albeit without the metadata info,) and decides they don't need to remove those mods from the real /Data directory and install them into MO. That could lead to plenty of unpredictable and undesired results.So, a warning would be a very good idea, I believe.

Edited June 3, 2014 by keithinhanoi

[z929669]

@Keith

Sure, a warning is a good idea. I should be more specific I suppose ...

 

I am advocating special treatment of mods within /Data/. If MO can be made to automatically decipher mod names based upon their plugins, that's fine, but it is not necessary. What I really want to see is MO special treatment of the USP content as if it were official content. If this can be accomplished by auto-detection based on file name (or CRC check), fine ... but for the love of Whatever, please allow MO to recognize and specially handle USP content such that it is treated as if it were Bethesda content located within /Data/

 

This resolves our issues. If this treatment can be applied to ANY mod within /Data/, that is fine by me, and it can be associated with a warning (and also using special font styling). It will allow us the flexibility required to achieve the desired end result and possibly apply priority 0 rules to other mods as well (and it should be made very obvious that using a 'dirty' /Data/ should be considered Taboo and that the USPs are exempt as 'dirt' ... meaning that nobody should ever install loose assets within /Data/ and that any plugin/BSA mods installed within this location is acceptable, but not necessarily recommended practice).

 

My wish list:

• I want MO to inherently know how to prioritize plugins and assets associated with the official content and the USPs (priority 0, however Tannin wants to implement ... location within /Data/ seems obvious).
• I would also be perfectly happy with the idea of 100% coupling asset prioritization with plugin prioritization (as long as decoupling by mod [and globally] is possible). This makes for a very clean and organized and largely automated mod setup.
• I would also like to see clear GUI indicators of 'special' mods (see #1) and mods that have coupled versus decoupled prioritization (see #2)

[hauntmedoitagain]

Maybe I'm just missing something, but cant you just leave BSAs both unextracted and unchecked in the archives tab?

 

That would make the load order:

 

1. Official BSAs.

2. Third party BSAs loaded via plugin order.

3. Loose files.

 

I mean, that's how I've always ran things and I don't recall any issues.

[fireundubh]

What I really want to see is MO special treatment of the USP content as if it were official content.

Again, this is not necessary and Tannin has already said no to the idea of an inclusion/exclusion list.

 

It is very possible to read the first 9 bytes of every ESP very quickly to determine whether the ninth byte has the ESM flag. In other words, MO can auto-detect ESPs flagged as ESMs and treat those plugins accordingly. To demonstrate how easy this can be accomplished, in Ruby, for example:

#!/usr/bin/env rubyrequire 'hex_string'ARGV.each do|plugin|    esm = File.read('#{plugin}')[8].to_hex_string    if esm == '01'        puts '#{plugin} is flagged as an ESM.'    endend

The question then becomes, well, what do you do with them once they've been detected? Tannin doesn't want MO to muck around in the Data directory, so that leaves out moving files. Copying files ruffles the feathers of disk space conservationists. And symbolic links cause LOOT to fail. If MO can't manipulate those plugins in any way, what's left?

 

To make matters worse, there isn't a way to determine plugin-BSA relationships. How can MO know that AAA.bsa belongs to AAA.esp? File name? Well, there's no Skyrim.bsa for Skyrim.esm. And, for a time, Beards Unleashed and Beards had the same plugin name, so what would happen if the user had installed Beards Unleashed over Beards, overwriting Beards.esp as a result? If the latter mod had a BSA, it'd then be associated with the wrong plugin. File name comparisons is an easy yet unreliable way to validate relationships.

 

If MO had the capabilities of TES5Edit, MO could grab a list of the assets, along with their paths, used in each plugin and then try matching that list against the file list in each BSA. Despite the amount of work Tannin would need to put in to read plugins, even that method isn't a surefire way to accomplish the task. BSAs always contain the file names of packed assets and usually the paths, too, but not always. So, we're back to comparing file names. If BSAs included checksum data, MO could make this method work, but BSAs don't.

 

And I'm not even touching on how third-party content in the Data directory doesn't necessarily come neatly packaged in ESP/BSA pairs. Remember the days when people extracted loose files into the Data directory? Well, they still do.

 

Therefore, the best available option, in my opinion, is for MO to do nothing with those plugins and BSAs. Absolutely nothing. Force MO to ignore all third-party content in the Data directory and then tell the user what's happening so that they're not completely clueless about why they can't find their mods.

 

In the end, I think the recommendation to new MO users transitioning from NMM/Wrye Bash should be: reinstall Skyrim and leave the Data directory alone!

Edited June 3, 2014 by fireundubh