Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[SkepticalJoker]

I just want to testify here real quick.  After reading through this thread the other day, I decided to try undoing all of the BSA unpacking that I had done just to see what noticeable differences there were.  It ended up being somewhere around 150 mods that I had unpacked.  After I was finished reinstalling and leaving in the original BSA, I was pleasantly surprised to discover significant performance improvement in my game...much more than I expected.  Loading times are a mere fraction of what they were with loose files, and all of the stuttering that I had when loading cells has completely gone away.  Reverting to the bsa's was the only change I made to the game.

 

A few days ago, I was sure that unpacking BSA's was the way to go, but I have seen the light.  From now on, no more unpacking BSA's without a good reason.  

 

Yep, over the last couple days I've been repacking my texture mods and the HD DLC textures into BSAs, as well as packing originally loose file mods into new BSAs. My performance has increased greatly, stuttering has all but disappeared, load times are still reasonable, and crashes are less frequent. My HDD isn't great (Seagate Barracuda), so perhaps that's the issue. If I had an SSD or maybe one of those WD caviar blacks, BSA vs loose file wouldn't be much of a difference.

[DoubleYou]

I will say that Mod Organizer is definitely faster dealing with BSAs than loose files. I had the HRDLC extracted and optimized and whenever I'd check them, it would almost freeze MO before it got it all done. With it in a BSA, it was instantaneous. I have never observed this affecting in-game performance, however. I also think that speed is practically a mute argument in this debate.

[SkepticalJoker]

I think I'll go with TheBlokes proposal (the original one, see below) for now. It works and is easier to implement than my own, which means I will have a working solution quicker.My own proposal should still work and I might implement it in a future major update. This gives us more time to consider the implications.I'm still opposed to the idea of treating the unofficial patches special because while it's true that it's unlikely there will be more, there might be further mods from other authors that require similar treatment. Any solution should be as generic as possible, at least in the core program, I have fewer concerns with less genericity in plugins though:

 

Excellent. The Bloke's idea sounds great to me.

 

I do have to say that I don't understand the reluctance to treat the Unofficial Patches as special. They ARE special. They fix the game. You need them if you want anything approaching a reasonable expectation of completing the game. They also fix plenty of bugs that aren't game breaking per se, but severely diminish the quality of the game in any case.

 

As far as I know, no other mods do this. Other mods add things to the game, or make subjective changes to it. No other mod simply makes the game playable. I see no contradiction in treating the Unofficial Patches special, while not doing the same with any hypothetical future mods that want to use the same methods as the Unofficial Patches. To be honest, I don't see how any mod could do that without conflicting with the Unofficial Patches in the first place.

[DoubleYou]

A good argument for not treating the Unofficial Patches special would be:

What if the Unofficial Patches change form, and Tannin is gone, and we have it hardcoded to treat the Unofficial Patches differently?

 

None of us would desire that to happen.

[SkepticalJoker]

That would be a problem, yes. I don't know how likely that is, though? Also, the USKP team spent months debating the original change, so I imagine if the USKP team wanted to switch back for some unknown reason, we would have ample warning.

[z929669]

... snip/One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop.What do you guys think: Is this a good idea? Unfortunately it would not be possible to discover loose files associated with the foreign-installed esps, nor would I be able to figure out meta information about those mods (i.e. version, nexus id)

I very much like this idea. It can only help.

 

I also don't see any major reason why you could not eventually implement your original idea (all mods, loose and BSA sort with plugins), especially if we have the option to turn off this feature at the mod level (and globally too would be nice). Action only on mods with BSAs does not make sense, since any mod could (and maybe should) bepacked within a BSA. What is and is not affected would then be arbitrary and nonsensical. I favor the original idea of prioritizing ALL mod assets with plugin prioritization as a toggleable feature. This is a feature ADD, not a detraction.

 

Also, see my previous post about differential text/background colors of mods managed this way versus not in both mod and plugin lists. This way users can tell what list prioritization is relevant to mods in the corresponding lists.

[fireundubh]

One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop. What do you guys think: Is this a good idea?

You don't need to do that because the issue at hand only reveals itself when ESPs are flagged as ESMs.Just read the first nine bytes of an ESP and check the last byte. If the last byte is 01, that's a false-flagged ESP. 

@noobzor, thanks for confirming this, but I would also point everyone to a nice quote from Ethatron: I believe a couple SSDs are now passing the 500MB/s second mark for SATA 3 and SATA Express is also starting to show up in a few enthusiast/professional products, but that still will not make up the time gap until SSDs are in the +1.2GB/s range.

Samsung 840 EVO SSD has a 540 MB/s sequential read speed.DDR3 SDRAM has a 6400 MB/s maximum transfer rate.zlib 1.2.8-6 has a 300 MB/s decompression speed.As this data suggests, the file type matters though.What we need is a decompression test for BSAs. Edited May 30, 2014 by fireundubh

[EssArrBee]

You don't need to do that because the issue at hand only reveals itself when ESPs are flagged as ESMs.Just read the first nine bytes of an ESP and check the last byte. If the last byte is 01, that's a false-flagged ESP. Samsung 840 EVO SSD has a 540 MB/s sequential read speed.DDR3 SDRAM has a 6400 MB/s maximum transfer rate.zlib 1.2.8-6 has a 300 MB/s decompression speed.As this data suggests, the file type matters though.What we need is a decompression test for BSAs.

That article is from a Core 2 Duo laptop though. RAM doesn't matter at this point because you aren't bringing up the entire BSA to system memory and unpacking only what you need. Also, you can't move 3400MB/s anyways because the Skyrim engine would just laugh at you then explode. The CPU memory controller pulls the assets from the archive on disk memory and then has to move them from DRAM to SRAM to unpack (since the CPU has to run the actual algorithm). Then it gets moved again back to DRAM so it can move onto the GPU (if it is something GPU renders) or stays in SRAM and goes back to the CPU to be processed (scripts are an example of that). That whole process is usually seen as just one time linear time interval depending on size of the data. DRAM or SSD speed can't make the algorithm run faster, the CPU still has to put in the work. I think Ethatron used zlib as an example, but there are other compression types with way higher decomp times and that second test was with a an i5 @2.7GHz. I'm using an i5 @3.4GHz ATM so I'm going to get almost 400MB/s with zlib.

 

But, since BSAs might not even use zlib, yes, I agree we need a test for BSAs.

 

And, yes we need a BSA codec test.

[keithinhanoi]

 

One more question: Technically it would be easy to discover all esps (plus associated bsas) in the data directory and display them in MO as "special" mods. This would not only make vanilla skyrim + dlcs show up but also other mods installed outside MO, i.e. mods installed through steam workshop. What do you guys think: Is this a good idea?

You don't need to do that because the issue at hand only reveals itself when ESPs are flagged as ESMs.

 

Just read the first nine bytes of an ESP and check the last byte. If the last byte is 01, that's a false-flagged ESP.

Actually, if a user has installed ESP+BSA paired mods in the "real" <skyrim>/Data directory, then MO will place their BSAs at the top (earlier) in the BSA priority list. If MO doesn't pull them in as "special" mods that show up in the left-hand (middle) mod priority/"install" list pane, then it's still possible to subvert the BSA load order as compared to plugin load order.

 

If MO were changed so those mods were brought in as "special", allowing priority/"install" placement to set load order of BSA assets (but not loose files - which would be impossible,) then the user could move those mods in the priority/"install" list so that it matches load order, if they wish.

 

Possible issues resolved by doing this are vanilla-origin scripts/meshes/textures that the mod expects to overwrite by virtue of the game's normal BSA-tied-to-plugin load order. It also becomes even more important when thinking of the UPs, because if a user has any mod in the real /Data directory now which has BSA assets of the same names as anything supplied by the UPs, the UPs version will overwrite the ones of the other mod.

 

No - I think one of two things would be a very good idea here. Either:

[*]Tannin's suggestion of bringing all plugins (and their BSAs, if they have them) into the MO priority/"install" list as "special" mods, or

[*]a warning message that third party plugin-based mods were found directly installed in the real /Data directory, with a summary of the potential issues caused by that, but NOT bring those mods into MO.

Personally, I like #2 more, because really the whole point of MO is not to install mods directly into the real /Data directory, and if any of those mods included loose files assets (not allowed by Steam Workshop, right?) then it leads to further potential problems (and general confusion.)

 

Samsung 840 EVO SSD has a 540 MB/s sequential read speed.

 

DDR3 SDRAM has a 6400 MB/s maximum transfer rate.

 

zlib 1.2.8-6 has a 300 MB/s decompression speed.

 

As this data suggests, the file type matters though.

 

What we need is a decompression test for BSAs.

Not everybody has a machine with a fast processor, RAM, bus, SATA III, an SSD with on-the-fly firmware-based write compression.

 

So, please if any tests are done, include test systems with lower-end specs.

 

Though I don't know for sure, it should also be noted that with the STEP lineup there's likely a mix of BSAs with compression and no compression - no compression being a choice by mod authors because BSA compression doesn't seem to work with audio (and other?) file assets.

Edited May 31, 2014 by keithinhanoi

[fireundubh]

Actually, if a user has installed ESP+BSA paired mods in the "real" /Data directory, then MO will place their BSAs at the top (earlier) in the BSA priority list. If MO doesn't pull them in as "special" mods that show up in the left-hand (middle) mod priority/"install" list pane, then it's still possible to subvert the BSA load order as compared to plugin load order.

Sure, but the entire point of using MO is so you don't do that. Since MO now intercepts Steam Workshop downloads and puts them into Overwrite, there should be no reason to put anything in the Data folder. If you want to protect users from themselves, I'd more readily suggest that MO IGNORE all plugins in the Data folder except the core Skyrim and DLC plugins.

[keithinhanoi]

Sure, but the entire point of using MO is so you don't do that. Since MO now intercepts Steam Workshop downloads and puts them into Overwrite, there should be no reason to put anything in the Data folder. If you want to protect users from themselves, I'd more readily suggest that MO IGNORE all plugins in the Data folder except the core Skyrim and DLC plugins.

Um, maybe I didn't say it clear enough, but I actually agree with you.My suggestion though, is NOT to bring in third party mods from /Data, but give a clear warning message about their presence (in case the user wasn't aware.) see my #2 idea above - I edited it a bit to be crystal clear.

[DoubleYou]

Let me put it this way:

 

If MO imports Dawnguard, say, and leaves a mod that modifies Dawnguard with a BSA in the real data directory, the mod's assets will be overwritten by Dawnguard. Therefore, Tannin not only should import all mods from data, but also must in order to maintain compatibility in some cases. Certainly, in a perfect world where all users leave their data directory clean, this wouldn't be necessary, but Skyrim isn't Utopia.

[fireundubh]

edit: I ran Binwalk on the DLC BSAs. Looks like the BSAs use zlib compression. What version? Beats me.

 

edit 2: Okay, some BSAs use zlib compression; others use just LZMA.

Edited May 31, 2014 by fireundubh

[Tannin]

@fireundubh: whaaaat? To my knowledge BSAs only support zlib compression. Do you have a link to a bsa that uses lzma compression? MO wouldn't be able to decompress that.

[fireundubh]

@fireundubh: whaaaat? To my knowledge BSAs only support zlib compression. Do you have a link to a bsa that uses lzma compression? MO wouldn't be able to decompress that.

Lzma is the same compression used by 7-Zip. MO extracts .7z archives all the time.

 

Binwalk is a complex binary analysis program. Using its signature analysis feature on the core BSAs, the results suggest zlib headers for some while others are strictly lzma.

 

Some results are false positives. For example, Binwalk finds Ubicom firmware headers in "Skyrim - Misc.bsa."

 

However, of the 113 results, only three are Ubicom firmware headers whereas the remaining 110 are lzma compressed data. No zlib headers are suggested.

 

Since the signature analysis scans the whole file by default, these results could mean that .bsa is lzma compressed data or the .bsa is uncompressed but contains lzma compressed data.

 

In any case, I just found this program yesterday and spent the whole day trying to install Binwalk on CentOS, then installing Debian, and then sorting out dependency issues. So, I'm no expert.

 

Here's what Update.bsa looks like through the lenses of signature and entropy analysis:

 

Edited May 31, 2014 by fireundubh