Ramifications of BSA Extraction in Mod Organizer

[z929669]

This will graduate to wiki guide format once we get sufficient community input ...

---

 
I am creating this thread to address some important information about BSA extraction and related modding concepts that come up over an over again with respect to STEP's use of Mod Organizer (MO) and MO/STEP critics that disagree with STEP's general advocacy of BSA extraction and MO's BSA-extraction functionality. In order for this to be meaningful to all users, I am providing some background information required for understanding the ramifications of BSA extraction and why STEP advocates it while other respected modders do not.
 
WARNING: Extracting BSAs deviates from the intent and expectation of the respective mod provider, so it is fair to expect that once a user extracts a BSA supplied with a given mod, that mod's author has every right to refuse support on the basis that the user implementation is no longer the same mod. Further, if a user extracts vanilla Skyrim BSAs, ALL mod authors and the modding community at large have the right to refuse ANY support to said user!
 
NOTE: The STEP community follows Wrye's cathedral model of modding, and since we advocate experimentation and an open modding approach, we also try to encourage 'creativity' (but not flagrant stupidity, TYVM) and users are welcome to support each other in this community, and we'll do our best to help as we are able, particularly with regard to what we advise in any of our guides. After all, modding is a largely creative endeavor and a learning experience!

---

 
Executive Summary
 
STEP provides instruction for a range of methods to mod Skyrim. Novice users can follow the STEP Guide, which is pretty straight forward and adheres to conventional modding methods. More advanced users or users that want to take it further can follow the additional guides and employ some more advanced and less conventional techniques (like BSA extraction and texture optimization) that we have found to be viable and optimal for the 'perfectionistic' or 'adventurous' modder. Some of these techniques are traditionally the jobs of the mod providers rather than the mod users, but we encourage and empower our community to take on these tasks if they wish, because many mod providers do not optimize their mods, and we advocate customization of mods by the user.
 
STEP advocates BSA extraction because it allows for more granular control of the modded Skyrim (but see 'cons' below). Critics object that this practice 'breaks' the fundamental standards of install-order and load-order methodology that many mods and modding utilities are built around. Nevertheless, I and many STEP staff and members have not found this to be the case and propose that these concerns likely stem as a result of Mod Organizer users having problems and complaining and blaming the mod authors on mod threads like the USkP when something goes wrong with their game. Why MO users? Because MO is the only mod manager that exposes BSA extraction to any user that installs a mod with a BSA in it (basically, all MO users).

• STEP advocates BSA extraction (given the user understands the ramifications!)
  • It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)
  • Used properly, it does not cause any load-order issues at all
  • It allows more granular control of a modded setup
  • It can theoretically lead to excess disk fragmentation (HDDs only, not SSDs)
  • It can theoretically reduce in-game performance if most/all game assets are loose files under a heavily-modded game
• BSA extraction is not 'bad' (NOTE: it can alter the intended behavior of a mod's interaction with other mods if used improperly)
• BSAs are not 'bad'
  • simplify mod distribution
  • simplify user maintenance (NOTE: particularly for manual installation and traditional mod management ... but NOT for Mod Organizer users!)
  • simplify support by mod providers
  • BSAs are the future trend in Bethesda modding, so best to get used to them

---

 
Install Order vs Load Order
 
A word about game assets: Game assets are any files (resources) that get used by Skyrim. Plugins (*.esm & *.esp) are special assets that are used by Skyrim to call upon other assets and to provide instructions for their use. Thus, plugins are the 'brain' of the game. The only plugins needed to play Skyrim are Skyrim.esm &  Update.esm. Other plugins are from the DLC add-ons or other mods made by the modding community like the USkPs, etc.
 
Install order: The order of mod files installed onto disk. If two mod packages contain the same file names along the same file paths (e.g., textures/blah.dds), then the last installed version overwrites (i.e., overrides) any previous version and is thus the version that will be used by the game if it is called upon by the game (via a plugin). These mod files can be anything at all (e.g., text, images, ... whatever), but only certain file types are used by the game: textures, meshes, scripts, plugins, etc. Therefore, install order affects what resources the game will use.
 
(NOTE: MO does not install mods to the data directory, but rather mods are extracted into mod folders within a user-specified location. MO creates a virtual /Data/ that appears to Skyrim as the actual /Data/, and it populates this virtual directory with mod assets from the install directory as specified by the MO installation priority. Otherwise, there is effectively no difference between MO and other mod managers, but this difference is fundamental and confers a significant advantage to MO users).
 
Load Order: The order that plugins are loaded into the game. Like install order, the last plugin loaded overrides all previous plugins. Since plugins reference assets within /Data/ by file name, there is potential for two different plugins to reference the same named resource.  Additionally, since plugins provide instructions as to the use of these resources, load order can also affect game behavior. Therefore, load order affects both what the game will use and how the game will use it.
 
What are BSAs?
 
Mandatory reading: read this important background information!
 
BSA: A proprietary archive of game assets that mirrors /Data/ directory structure. Thus, a BSA file is an archive exactly like a folder that is simply packaged as a file. The same is true of any ZIP or 7z archive.
 
How do BSAs Work?
 
For Skyrim to be 'aware' of a BSA, it must either be registered in Skyrim.ini or loaded with a plugin of same name. Once recognized, the game sees any BSA as part of /Data/ itself; however, when conflicts exist between files contained within a registered BSA, a plugin-loaded BSA or within /Data/ as loose files, things are a little trickier:

• Registered BSAs: These load at Skyrim start in the order that they are listed in Skyrim.ini, last loaded BSA 'wins' in event of resource conflicts of contents within.
• Plugin-loaded BSAs: These load when a new or saved game is loaded after Skyrim starts. Each BSA is loaded at the time the plugin of same name is loaded. So any BSA with content resource conflicts corresponding to a plugin will 'win' if its plugin is loaded after the conflicting plugin. Basically, these BSAs (and all of their asset content) are referenced by their plugin and loaded according to plugin load order. Plugin-loaded BSAs always 'win' where they conflict with Registered BSAs. The only exception is with respect to resources required at Skyrim start but before savegame (or new game) load, like No Menu and Loading Smoke.
• Note about loose files: Loose files always override same files inside of registered AND plugin-loaded BSAs!

 
Summarizing in terms of prioritization and load order ...
 
Skyrim asset priority:

1. Loose assets always win
2. Plugin-loaded BSAs win all but #1 (EXCEPTION: plugins are only loaded when a new or saved game is started, so plugin-loaded BSAs have zero priority with regard to pre-game assets)
3. Registered-BSA assets lose to all #1 & #2
4. Registered Skyrim BSAs and other official content and DLCs behave no differently than "after market", mod BSAs

Plugin/BSA load order:

1. Registered BSAs load according to list order in Skyrim.ini
2. Plugin-loaded BSAa load with respect to the corresponding plugin load order
3. Plugins load according to %USERPROFILE%/Appdata/Local/Skyrim/plugins.txt, which is managed by BOSS/LOOT

BSA Pros:

• Keep the Data directory clean and uncluttered (NOTE: this does not apply to MO users though, since MO uses the virtual file system).
• Allow easy mod management, since all of a mod's files are much simpler to identify and update or remove (mitigates user error= less support burden)
• Make it easier for mod authors to distribute and maintain control over how the mod functions (mitigates user error = less support burden)
• UPDATE:
• Better performance (NOTE: a lot of loose files slows down game startup, especially when using MO)
• Less disk usage (NOTE: BSAs can be compressed; HDD fragmentation is less of an issue)

BSA Cons:

• Removes an element of user-level control ... and many mod users are control freaks (STEP especially)
• Users can no longer efficiently see contents of a mod (NOTE: although Wrye Bash does expose this information, albeit with a performance hit ... is this functionality inherent or is it off by default??)
• Incentivizes mod authors to provide BSA 'hotfixes' as loose files (NOTE: This has undesireable ramifications for MO users due to behavior of BSA extraction in MO ... BSA extracts last, so loose file hotfix is overridden by original version within the BSA! EDIT: this is fixed in the current beta and next release of MO)
• Mod authors are forced to upload all files (the entire BSA) for any updates (all files are contained within a BSA), and users are forced to either download again or deal with the issue just previous if the mod author has supplied a 'hotfix'-type update.

BSAs & Steam Workshop
Steam Workshop only allows mods that use the BSA + ESP format. STEP finds this overly restrictive and unnecessarily 'controlling'. I personally resent it and only deal with Steam because it is the wrapper for Skyrim (unfortunately, IMO). The Steam Workshop and Steam-Skyrim community are valid entities that do not deserve to be totally ignored, but STEP does not recommend that it be used as a primary source for mods or modding information. The Nexus is the STEP-preferred source for all modding needs. For information, STEP is a good primary source, and we point to the best alternative sources, but here are a few others:

• TESAlliance
• Bethsoft Forums
• Unofficial Elder Scrolls Wiki (UESP)
• SkyrimSearch.com
• Tamriel Maps 

BSAs & Mod Organizer
 
Since Mod Organizer allows users to extract BSAs during mod installation, MO potentially obviates any functionality of registered or plugin-loaded BSAs. Thus, any mod that uses a BSA is effectively constrained henceforth by rules pertaining to loose files, so its assets are no longer linked to hierarchies of BSA registration order or plugin load order. This and the fact that all or user-specified mod resources can be loose and manageable by MO confers a clear advantage to the user.
 
BSA Extraction Pros:

• MO users have a much more granular level of asset control and can prioritize BSA contents at the loose-files level
• It is a prerequisite to texture optimization (NOTE: repackaging vanilla BSAs is possible with CK's Archive.exe but problematic with BSAopt)

Other issues can arise though, so only informed users that understand the ramifications should be using this functionality (unpacking BSAs). Following are some things to be aware of when unpacking BSAs (that mod authors intended to remain packed as delivered!).
 
UPDATE: There does not seem to be any need for standard users to extract mod BSAs in MO, because once can subvert the constraints of the standard load order/asset prioritization system from the Archives Tab:

• Plugin checked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin checked, BSA unchecked - Follows plugin load order for all unchecked BSAs. All loose file assets will "overwrite." OTHER checked BSAs will NOT overwrite, which is why unchecking BSAs can lead to unpredictable results or dificult-to-resolve conflicts (hence the warning).
• Plugin unchecked, BSA checked - Follows mod priority order for conflict resolution. Plugin does not affect the situation at all.
• Plugin unchecked, BSA unchecked - As if the plugin and BSA don't even exist in the mod setup.
• Furthermore, MO will scan all mod BSAs (aside from those in /Data/) and include these assets in the Mod > Information > Conflicts tab. So in MO, BSAs effectively behave like loose files when checked in the Archives Tab! Mod developers will still find the BSA extraction functionality handy for testing purposes during production of updates to their existing BSA-packed mods or when developing new ones dependent on assets contained within BSAs. 

BSA Extraction Cons:

• BSA assets are now given loose files priority, so this alters the mod author's original design intent and may introduce false 'bugs' that nobody on any forums will likely want to or know how to diagnose or fix ;)
• BSA extraction in MO happens after loose files are installed. This means that any loose 'hotfixes' would be overwritten by the BSA version, which is outdated. EDIT: this is fixed in the current beta and next release of MO

MO exposes BSA extraction functionality using a prompt when a mod containing a BSA is first encountered. This functionality can henceforth be "always allowed" or selective, based on user preference in response to this prompt. Users that do not fully understand the ramifications of BSA extraction on the specific mods they are using together should not use this feature. If "automatic BSA extraction" is in effect, it can be reset from Settings (click the wrench icon in the toolbar) > Reset Dialogs > click 'yes' at the prompt.
 
First, STEP recommends that users NEVER "always allow" automatic BSA extraction ... why? Because there is no need to do this at all, since the granular functionality already resides within the Archive Tab. More importantly, because many unknown or unintended prioritization issues can come into play as described previously. It is always safer to use the BSA unless it will cause a de facto undesirable result.
 
BSAs that have optimized textures and can be overridden completely by downstream mods should stay inside BSAs (or repackaged using CK's Archive.exe). If assets from inside a BSA need to overwrite some mods and be overwritten by others, then sometimes it makes sense to extract the BSA. MO has a beautiful tool accessible from within its Archive Tab. If a BSA is present in the load order, it will appear in the Archives Tab. Leave it unchecked to allow it to behave normally and be loaded by its plugin (if the plugin is active), and check it to extract the BSA to the mod folder and effectively confer loose-files prioritization.
 
Critics of Mod Organizer and STEP (for Officially Advocating BSA Extraction) 

Some within the well-respected modding community are at odds with the idea of BSA extraction advocated by STEP and facilitated by MO. The most notable contingent is the USkP team. This is relatively old news and nothing that should be shocking, so please do not treat it that way. The reasons are not unfounded and actually valid. I bring this up solely to address the idea to remove BSA-extraction from MO that Tannin suggested if MO-detected load order issues are not resolved properly (by submitting a ticket). In fact, I created this thread to address this one issue as much as to address the knowledge gap that is the real cause of any issues associated with BSA extraction.
 
Some modders are more or less happy with MO's ability to invoke BSA extraction. Generally, mod authors who have gotten a lot of grief with respect to their mods --for problems caused by the ramifications of rampant BSA extraction-- seem to have more of a problem with MO (see note below!). This has been somewhat problematic for STEP and MO with respect to outsiders privy to the argument but not privy to STEP or MO ... never mind that the 'fault' should be shouldered solely by the unwitting mod user for invoking BSA extraction without understanding the ramifications of doing so ... and ours for not properly educating our user base to that effect (hence this thread).
 
Let it be said that the STEP modding community and the vast majority of modders are the kinds of users that use PCs instead of Macs and tend to be somewhat removed from computing 'norms' imposed by Apple and Microsoft and their ilk. In general, we do not like our control restricted in favor of provider control over our resources to make their lives simpler. We are generally in favor of digital freedom, open source software, and the honor system. Big Brother and his methods are generally unwelcome. Demonizing BSA extraction in general and removing it from MO in particular in order to enhance level-of-conrtol by mod providers would be a big mistake, as STEP itself somewhat relies on this feature (and will to a larger extent in the future). However, I think that it is very important that our users understand the ramifications of using BSA extraction, and we need to address explicitly in the STEP Guide.
 
The MO user base (not its 'critic' base) should guide Tannin's direction of MO, IMHO ... 
 
I want to explore what we do (and do not) know with regard to the ramifications of BSA extraction to our MO users and how best to make MO a broadly accepted utility among all within the modding community ... not just STEP. In order to do so, we must not dredge up combative arguments. Constructive argument is good for all respective modding endeavors, so what has been said in the past is water under the bridge. So keep it lively and fact filled ... but keep it polite and considerate!
 
Please report bugs as Tannin requests using the link above. Also please post to this thread and help us to improve the breadth and accuracy of this OP!
 

[keithinhanoi]

Symbolic links.

 

 

Yes, yes, I know about symbolic links. Been using them in MacOS, *nix, and Windows for several years, including space-saving tricks when I had a smaller SSD + a large HDD. I use them so often I even installed this software, which allows you to make symbolic links using the contextual menu in Windows Explorer. One right click to pick the source, and then another right click to drop the symbolic link in your desired location. Cool stuff.

 

But - no, my second suggestion with moving the UPs BSAs out of their MO mod folders into the real /data directory won't work with symbolic links, because of that operative word... move.

 

Symbolic links are for making a "doppelganger" file or directory in a different location, that the system will treat just like the original.

 

MO will only move the UPs BSAs into the "/data" group in the assets Priority tab if their BSAs are located in the real /data directory and not in the MO mod folder.

 

Explained another way: There can only be one copy of each BSA being used, in the real /data directory, for it to show up in the /data list in the Priority tab (of the right-hand pane).

 

So the symbolic link method can only be used as a more elegant way of making MO see the official DLCs as "mods" in the left-hand mod priority/"install" list pane. And yes, using symbolic links will preserve the pristine /data folder so that the Steam validate archive feature works, and updates work, etc.

 

But I imagine where it seems like an elegant solution to me, there's probably some situation I'm not thinking of where a user could completely screw it up.

 

No, its actually worse than that because a "managed" BSA (let's just call them that) can overwrite loose files if it has higher priority but not a priority-loaded BSA.

 

So then if one BSA is not "managed" (un-ticked/un-checked) it goes like this:

[*]Registered (official) BSAs

[*]All managed (ticked/checked) BSAs -and- loose files, loaded according to MO's mod priority/"install" order

[*]All un-ticked/un-checked BSAs not managed by MO, loaded according to their corresponding plugin order

or if it's more convoluted than that, then I think on this one I just can't wrap my brain around it, sadly.

 

For some reason (one I never understood) the update.bsa appearing in the list confused some people - a lot - so I hid it. the update bsa is loaded by the engine hard-coded so it's a completely separate thing.

 

Hmm, I didn't know that. I can't fathom why it would confuse people. But as long as we know it's correctly loaded after the original vanilla skyrim BSA assets, then fine.

 

No, its actually worse than that because a "managed" BSA (let's just call them that) can overwrite loose files if it has higher priority but not a priority-loaded BSA.

 

For some reason (one I never understood) the update.bsa appearing in the list confused some people - a lot - so I hid it. the update bsa is loaded by the engine hard-coded so it's a completely separate thing.

 

I could still show the conflicts but it would be rather pointless because loot/boss would then be responsible for putting bsas into the right order and loose files would always overwrite bsas so you really wouldn't have to worry about conflicts with bsas (nor would you be able to do anything about it)

 

If you do decide to complete get rid of BSA priority management as it stands currently in MO, I sincerely hope you'll consider keeping the conflict reporting at least for loose files.

 

Better yet, let MO continue to show asset conflicts that come through from the plugin-loaded BSA assets - because despite not having control over it directly in MO, it could be handled through LOOT/BOSS user rules to change plugin load order, or by extracting the assets desired in the game and throwing those loose files into a separate mod (so the loose files "win" over the BSA loaded versions.)

 

I have to say, being able to hunt down assets provided by two different mods in the way the MO does it is a complete godsend.

 

Anyhow, I've made my two suggestions, as has been nicely summed up by TechAngel. Shall I make them official suggestions on the MO Bug Tracker site, Tannin?

Edited May 29, 2014 by keithinhanoi

[Tannin]

@z: false esm aren't the problem, the problem is that these unofficial esms are supposed to be loaded before official content. I haven't come across any mod for fallout 3/nv that expected to be loaded before official stuff.

 

Sorry guys, symbolic links / junctions are not an option.

- They require administrator rights to create, MO was explicitly designed to run with restricted rights.

- They don't work on FAT HDDs (admittedly those are becoming increasingly rare)

- Windows Explorer, at least in previous windows version, doesn't deal well with junction points (demonstrated - among other things - by the fact you can't create junctions inside the primary file management tool of windows). They are also not well understood by many users and third party tools. This just gives me a a vaguely bad feeling about the approach. Users might end up in a situation where they can't clean up properly without tools integrated in Windows or they might end up deleting files when they only intended to remove the junction points.

 

I also don't like the idea of treating the unofficial patches special because they are third party, unofficial content after all. There is no way of knowing if the problem ends there, there might be other mods that expect to be installed before the official content or before the unofficial patches. I might end up continuously adding exceptions for mods and if I ever stop developing MO things would deteriorate.

 

Treating the DLCs as mods is a far more feasible approach because the number of official DLCs is a lot more manageable than that of the unofficial stuff. TheBlokes proposal seems rather feasible but it means quite a bit of work for me and it still won't setup things correctly for users by default.

 

 

I came up with another idea but this is an even bigger change. But first the rationale:

Right now MO enforces the distinction between mod order=installation order=asset order vs plugin order. Its BSA management is there so bsas that would usually be plugin-loaded and therefore in plugin order (contrary to loose files that are in mod order) also become mod ordered.

 

I still think think it's reasonable to not have a different order between loose files and bsas.

 

However, in the vanilla system if there were no loose-file mods then in the default game there would only be one order to manage: plugin order. That's not bad either...

 

Could we transfer this to MO?

 

We could have all mods (that contain plugins) by default be "auto-storted". All auto-sorted mods are automatically put into the same order as the contained plugin.

I would also change the bsa list into a flat list that is also auto-sorted by plugin order. This means MO would no longer prevent bsas from begin loaded interleaved with those from other mods/the core game.

 

By default MO would then behave like the regular game: assets are loaded in the same order as the corresponsing plugin except this would apply to ALL assets including loose files.

This means by default the user manages only one order: plugin order - which he can delegate to LOOT. Therefore he will by default have a working install.

Of course he can still re-order mods that have no plugin (pure replacers consisting of loose files). He can still disable dummy esps. And if he really wants he can disable auto-sorting for individual mods to take control of their mod order, but then he will have to manually look after the bsa order for that mod as well.

 

This provides a system that should work well by default and can still be customized. Opinions?

[Tannin]

So then if one BSA is not "managed" (un-ticked/un-checked) it goes like this:

[*]Registered (official) BSAs

[*]All managed (ticked/checked) BSAs -and- loose files, loaded according to MO's mod priority/"install" order

[*]All un-ticked/un-checked BSAs not managed by MO, loaded according to their corresponding plugin order

or if it's more convoluted than that, then I think on this one I just can't wrap my brain around it, sadly.

 

I'm afraid it's more convoluted:

[*]Registered (official) BSAs

[*]All managed (ticked/checked) BSAs -and- loose files (1), loaded according to MO's mod priority/"install" order

[*]All un-ticked/un-checked BSAs not managed by MO, loaded according to their corresponding plugin order

[*]loose files (2)

[*](1) is loose files that conflict with managed BSAs or other loose files

(2) is loose files that conflict with plugin-loaded BSAs but not with managed-bsas. They will win against plugin-loaded BSAs no matter the order of either the mod or plugin but they won't win agains managed bsas of higher order even though the plugin-loaded bsas would win against the managed bsa.

[z929669]

@Tannin

That works for me (for STEP) as long as the auto-sort can be disabled and BSA extraction is still toggleable. LOOT/BOSS should never be given total control without opportunity to decouple user-specified mods. This could mess with some of our logistical mod installation order, so it would best be something that could be disabled at the mod level if at all possible (i.e., not ideal to have the all or none toggle for this behavior).

 

I assume all other non-disputed functionality would remain (e.g., conflict resolution of all assets, including BSA assets)? Seems like a good solution and keeps the Data directory clean as always. It also, by default, removes confusion of the degrees of freedom conferred by independent asset and load priority (it can get confusing for those that know about them!).

 

EDIT: Tannin, just saw you last post ... isn't it simpler to just think of managed BSA assets as loose files coupled to the Mod install order just as are loose files? I got confused reading that!

[fireundubh]

- They require administrator rights to create, MO was explicitly designed to run with restricted rights.

Junctions require only access to the file system, but then junctions work only with directories, not files. 

- They don't work on FAT HDDs (admittedly those are becoming increasingly rare)

Extremely rare to nonexistent as far as PCs go. 

Users might end up in a situation where they can't clean up properly without tools integrated in Windows or they might end up deleting files when they only intended to remove the junction points.

Symbolic links. When you delete a symbolic link, you delete only the symbolic link.Also, symbolic links have been supported in Windows since Vista. Unfortunately, you need a (free) third-party driver for XP. 

I might end up continuously adding exceptions for mods and if I ever stop developing MO things would deteriorate.

You could add an exceptions textarea/listbox to the settings window, but then you'd need to educate users about that textarea/listbox. 

I came up with another idea but this is an even bigger change. [...] This provides a system that should work well by default and can still be customized. Opinions?

I was trying to suggest things that minimized your workload, but if you're down to go crazy, have at it! ;) Edited May 27, 2014 by fireundubh

[fireundubh]

FYI: LOOT actually fails when you use symbolic links the way I suggested.

Error: Failed to calculate the load order. Details: Cyclic interaction detected between plugins "Unofficial Dawnguard Patch.esp" and "Unofficial Dragonborn Patch.esp".

[DoubleYou]

I also don't like the idea of treating the unofficial patches special because they are third party, unofficial content after all. There is no way of knowing if the problem ends there, there might be other mods that expect to be installed before the official content or before the unofficial patches. I might end up continuously adding exceptions for mods and if I ever stop developing MO things would deteriorate.

Which is why I like my solution. Allow an area for BSA custom sorting within a small, controlled place in the Archives tab. This will allow future similar problems to be simply dragged and dropped into the correct order. If you stop developing Mod Organizer, we can tell users what order the BSAs should be sorted in this place, and the need for this should be quite small. It would only need to be so for mods interleaved within unofficial content, which currently only applies to three mods: Unofficial Skyrim Patch, Unofficial Dawnguard Patch, and Unofficial Hearthfires Patch, but in future might include others.

 

The problem with automatic DLC detection and addition to the modlist is that the normal user won't, unless they read currently nonexistent instruction, interleave the Unofficial Patches with the DLC at the top of their modlist. They've had enough issues sorting the Unofficial Patches in the past.

 

The method Tannin suggests sounds like a feasible solution for the know-nothing modder. However, it would seem to me that this could add a great deal of volatility to the order of conflicts. You would almost have to add an "ignore sort order" button and a "lock mod order" button for the mods where this would mess up conflict resolution (I'm pretty sure there would be cases of this).

 

I still like my idea better, and it looks easier to me.

[Tannin]

@z: Of course there would be a way to customize the load order. There's also the userlist for loot so you can also influence plugin order manually.re your edit: that's pretty much the problem: You can't think of managed BSA assets as loose files if there is a plugin-loaded bsa. loose files overwrite plugin-loaded bsas, managed bsas can not.@fireundubh: sorry, yes, I was confusing junction points with symbolic links. But you're wrong about the access rights: According to wikipedia: "The default security settings in Windows Vista/Windows 7 disallow non-elevated administrators and all non-administrators from creating symbolic links."This can be changed, but I won't ask users to first modify their windows security settings before they can use MO.And MO is designed to be portable. You can put MO and all your mods onto a usb stick, take it to a friend, run your whole mod setup on his computer. And on usb sticks non-ntfs filesystems are still common.

I was trying to suggest things that minimized your workload, but if you're down to go crazy, have at it! ;)

crazy? I'm way past that. ;)

[fireundubh]

@fireundubh: sorry, yes, I was confusing junction points with symbolic links. But you're wrong about the access rights: According to wikipedia: "The default security settings in Windows Vista/Windows 7 disallow non-elevated administrators and all non-administrators from creating symbolic links."

Symbolic links require elevated privileges; however, neither hard links nor junctions do. Symbolic links, hard links, and junctions are treated as different things in Windows, although all are created with the same program.

And MO is designed to be portable. You can put MO and all your mods onto a usb stick, take it to a friend, run your whole mod setup on his computer. And on usb sticks non-ntfs filesystems are still common.

This sounds like needless complexity for an impractical application. Edited May 27, 2014 by fireundubh

[keithinhanoi]

I'm afraid it's more convoluted:

[*]Registered (official) BSAs

[*]All managed (ticked/checked) BSAs -and- loose files (1), loaded according to MO's mod priority/"install" order

[*]All un-ticked/un-checked BSAs not managed by MO, loaded according to their corresponding plugin order

[*]loose files (2)

[*](1) is loose files that conflict with managed BSAs or other loose files

(2) is loose files that conflict with plugin-loaded BSAs but not with managed-bsas. They will win against plugin-loaded BSAs no matter the order of either the mod or plugin but they won't win agains managed bsas of higher order even though the plugin-loaded bsas would win against the managed bsa.

Okay, well, thanks for explaining it. I've got my brain properly wrapped around this, but it still hurts thinking about it.

 

Sorry guys, symbolic links / junctions are not an option.

- They require administrator rights to create, MO was explicitly designed to run with restricted rights. <-- This is the actual show stopper

- They don't work on FAT HDDs (admittedly those are becoming increasingly rare)

- Windows Explorer, at least in previous windows version, doesn't deal well with junction points (demonstrated - among other things - by the fact you can't create junctions inside the primary file management tool of windows). They are also not well understood by many users and third party tools. This just gives me a a vaguely bad feeling about the approach. Users might end up in a situation where they can't clean up properly without tools integrated in Windows or they might end up deleting files when they only intended to remove the junction points.

FYI: LOOT actually fails when you use symbolic links the way I suggested.

fireundubh, a last option for a personal solution could be to try just making symbolic links for the official DLC BSAs only - and not any of the .esm plugins. But yes, I figure the whole symbolic link thing would be a no starter for the general masses.

 

This provides a system that should work well by default and can still be customized. Opinions?

So basically, what you are describing is MO doing what it does now with BSA management, but adding the default option of allowing BSA asset priority to be set by the plugin order - but those BSA assets are still loaded using MO's "black magic" rather than the game's native plugin-loaded BSA mechanism - right?

 

It's very interesting, but there are quite a few what if situations that I could come up with that I'm not sure how to answer with this concept as you've described.

 

First off, when you toggle BSA asset priority between user managed and plugin order determined, what happens to that mod in the left-hand mod priority list pane? Does it get moved around, or are we now going to manage BSA and loose assets in the Assets tab only?

 

Then, what if the mod in question has two BSA's - say for example Immersive Armors, which has a main plugin and BSA, and a secondary "dummy" plugin and BSA pair. Currently, I can disable the dummy plugin, but in your new scheme, in absence of the dummy plugin, I need to then manually manage priority order of IA's second BSA. It might be set by the mod author with the assumption that the second BSA loads after the main one, so I need to be aware of that and set the second BSA priority to a higher number (aka further down in the list for x to understand) than the main mod's slot. As an end user, I might also have extracted some textures to optimize them, and they could also be in the same MO mod folder for IA.

 

With the IA example it seems tricky, but then the best way to represent it visually in the GUI would have an assets management tab, which splits apart every BSA into a single line entry (to get around mixed managed and plugin-order determined BSAs, plus loose files contained in the same mod.) So with the IA example, you'd see this:

 

[+]  Hothtrooper44_ArmorCompilation.bsa    |    Immersive Armors

[-]  Hothtrooper44_Armor_Ecksstra.bsa     |    Immersive Armors

[-]  Loose files                                            |    Immersive Armors

[ ]  DeadlyDragons.bsa                                |    Deadly Dragons - Dragonborn

 

In this represenation, [+] is a plugin-ordered BSA (the default), [  ] is a user-managed BSA (un-ticked), and [-] is user-managed loose files or an "orphaned" BSA.

 

If the Assets tab becomes the place to manage asset priority order, then the left-hand pane would merely become the list of installed mods, and no longer be used to represent priority for assets loading. A different paradigm, but I can't think of any way to deal with the potential mixed situations as I've described with the IA example.

 

Now another question is about what happens when you turn plugin-order determined BSA load priority? Does that BSA in the list jump into it's place between two other corresponding plugin-order managed BSAs? What if there are some user-managed entries between them? Will MO put it at the top or bottom (closest to the next lower or higher numbered plugin-order managed BSA)?

 

Anyhow, like I said, the idea is very interesting, but it will require some considerations about how the GUI need to be re-implemented to accommodate the method.

 

[Tannin]

Which is why I like my solution. Allow an area for BSA custom sorting within a small, controlled place in the Archives tab. This will allow future similar problems to be simply dragged and dropped into the correct order. If you stop developing Mod Organizer, we can tell users what order the BSAs should be sorted in this place, and the need for this should be quite small. It would only need to be so for mods interleaved within unofficial content, which currently only applies to three mods: Unofficial Skyrim Patch, Unofficial Dawnguard Patch, and Unofficial Hearthfires Patch, but in future might include others.

But we'd now have 3 "orders" to manage instead of one or two. The system needs to be easy in the common case (which does include the unofficial patches) and complicated only in the special case (I rather want this leather armor texture than the other)

The problem with automatic DLC detection and addition to the modlist is that the normal user won't, unless they read currently nonexistent instruction, interleave the Unofficial Patches with the DLC at the top of their modlist. They've had enough issues sorting the Unofficial Patches in the past.

But the "Potential mod order problem" suggestion would tell you that the unofficial patches need to go between the dlcs!

The method Tannin suggests sounds like a feasible solution for the know-nothing modder. However, it would seem to me that this could add a great deal of volatility to the order of conflicts. You would almost have to add an "ignore sort order" button and a "lock mod order" button for the mods where this would mess up conflict resolution (I'm pretty sure there would be cases of this).

What you have to realize is that the conflict resolution loot does (which, in the vanilla state also sorts BSAs) decides game stability. It is important! The conflict resolution you can do manually, changing mod order on the left, should be purely a matter of taste. No mod consisting of loose file should assume to be installed in a certain position!The left pane in MO is for customization, there shouldn't be a "necessary" order to get your game running and if there are dependencies, MO should tell you and you should follow dependencies. The need for game stability beats the wish to customize.And therefore there wouldn't have to be a "ignore sort order" button. You would have auto-sorted mods and only for individual mods you could disable it if you absolutely know assets and plugin can be ordered independent of each other..This isn't good only for know-nothing modders, it's good for all of them. Customize only what you know can be customized

[Tannin]

Symbolic links require elevated privileges; however, neither hard links nor junctions do.

Hard links are even less of an option because they can only link within one drive. junctions aren't an option because they only work on directories. Symbolic links don't work because they require elevation.

This sounds like needless complexity for an impractical application.

Doesn't matter, it's one possible use of the tool we would lose, in addition to the additional complexity for the user plus the elevation problem -> not going to happen.

So basically, what you are describing is MO doing what it does now with BSA management, but adding the default option of allowing BSA asset priority to be set by the plugin order - but those BSA assets are still loaded using MO's "black magic" rather than the game's native plugin-loaded BSA mechanism - right?

right

First off, when you toggle BSA asset priority between user managed and plugin order determined, what happens to that mod in the left-hand mod priority list pane? Does it get moved around, or are we now going to manage BSA and loose assets in the Assets tab only?

This wouldn't be a supported option any more. We have already determined that having both managed BSAs and plugin-loaded BSAs is too complex. There isn't an asset tab, there is only a BSA tab which controls only the bsas on/off state, while their order (by default) would be determined automatically. And you wouldn't manage assets at all unless you specifically enable that for a mod and then you would use the same UI as you always have.

Then, what if the mod in question has two BSA's - say for example Immersive Armors, which has a main plugin and BSA, and a secondary "dummy" plugin and BSA pair. Currently, I can disable the dummy plugin, but in your new scheme, in absence of the dummy plugin, I need to then manually manage priority order of IA's second BSA.

You could still disable the dummy plugins, both BSAs would be loaded if they are checked on the Archives tab the same as they always have. You wouldn't manage their priority manually, that would still depend on the order of the plugins even if they are disabled. LOOT (and BOSS) does order disabled plugins!

It might be set by the mod author with the assumption that the second BSA loads after the main one, so I need to be aware of that and set the second BSA priority to a higher number (aka further down in the list for x to understand) than the main mod's slot.

No you wouldn't because LOOT is already aware of it. If it isn't, the BSA would be loaded in the wrong order for ALL mod managers, not only MO.

As an end user, I might also have extracted some textures to optimize them, and they could also be in the same MO mod folder for IA.

Not a problem, because MO would already sort mods in the same order as BSAs and plugins

In this represenation, [+] is a plugin-ordered BSA (the default), [ ] is a user-managed BSA (un-ticked), and [-] is user-managed loose files or an "orphaned" BSA

No, you won't be able to user-manage individual BSAs, either you manage the whole mod or none of it. If you want to extract a BSA you delete (or rename) it afterwards to avoid any confusion on what gets loaded (even though MO would ensure the loose file overwrite the BSA)

If the Assets tab becomes the place to manage asset priority order, then the left-hand pane would merely become the list of installed mods, and no longer be used to represent priority for assets loading. A different paradigm, but I can't think of any way to deal with the potential mixed situations as I've described with the IA example.

This is not going to happen. You either let MO manage the mod manually then the BSAs are ordered in plugin order and loose files within a mod overwrite the bsas of that mod and all of lower priority or you manage manually then you specify BSA ordering for that whole mod and mod priority manually and loose files within a mod will still overwrite the bsas of that mod and all of lower priority.In essence: For auto-sorted mods you only ever manage one list: plugin order. For user-sorted mods you manage three lists: plugin, archive, mod

Now another question is about what happens when you turn plugin-order determined BSA load priority? Does that BSA in the list jump into it's place between two other corresponding plugin-order managed BSAs? What if there are some user-managed entries between them? Will MO put it at the top or bottom (closest to the next lower or higher numbered plugin-order managed BSA)?

The bsa and the mod would follow the plugin order, user managed entries would simply keep their original priority.If your order isA (plugin managed)...B (user managed)...C (plugin managed)And then C is moved above A then the order might becomeCABor, if B was directly below A:CBA

[TechAngel85]

There is a lot of information being tossed around about this possible new method. I've mainly been reading and keeping up with the thread and watching how things develop; however, I have a very specific question regarding STEP and this new method.

 

If you look at the STEP Guide, installation order (left pane) of mods is very important for STEP to handle proper conflict resolution in order to get the desired outcome. This has been an very important part of STEP since day one. What would this new method specifically mean for this installation order? Having a plugin order (right pane) which controls the mod order (left pane) could essentially break the STEP Guide. Very specifically will the plugin order (right pane) move mods around in the (left pane)? If so, I can see potential issues with the STEP Guide as well as any user specific mod order to handle conflict resolution in a specific way

 

If this is true, you're basically saying we'd be able to manually sort specific mods when needed; however, if the left pane and right pane are tied together in a way that if you move a mod in the left pane its plugin in the right pane also moves, this will create yet another issue of plugin conflicts that would have not existed when then two panes could be managed independently. The only way around this issue would to do yet another step of creating a list for LOOT to order them in the proper way.

 

I could be completely misunderstanding this whole thing but if I'm not, the new suggested method would seem to create a lot of unnecessary issues for users who aren't just "plug and play" users. Which when you think about it, how often do most of these "plug and play" users remain only that? Very few I would think. Most of these users, more than likely, become more advanced users over time who would eventually run into the issues described above.

 

Long post to simply ask, will the plugin order (right pane) move mods around in the (left pane) by default? If so, will we be able to turn off this default behavior so they can be managed independently?

[Tannin]

@TechAngel85: Why does the step guide need to be installed in a certain order? Is there documentation on which conflicts step resolves in a certain order and why?

 

The installation order is only relevant if there is a conflict between two mods, otherwise the order there simply doesn't matter.

 

For the cases where two mods conflict and the order of the resolution is important (not down to preference), step should probably be very very careful proposing anything other than the plugin-order anyway. Think about it from the perspecive of the mod author: If he knows his mod contains scripts that conflict with scripts from other mods and thus may break the game HE will have to ensure users install his mod correctly.

He could either provide an installation guide and hope people follow it. In this case step should follow his suggestion.

Or he could package the scripts and other assets in a bsa so that loot does the asset ordering to ensure users get the mod installed correctly.

The latter will work for people who don't extract BSAs and don't use MO, which should be the majority. It should never be necessary to unpack a BSA (or overwrite its load order) to get a mod working! In this case step should not suggest users unpack the bsa and order it contrary to plugin order, that would basically be saying "we know better than the author".

 

This leaves mods where the install order is down to preference it may be possible to provide the right order through a boss/loot userlist provided by loot. OR step would have to tell users to deactivate auto-sorting for that mod and where to put it then.

 

Long post short: Yes, MO would then move mods around based on plugin order. And it should not be necessary to turn this default behaviour off as it IS the usual behaviour for esp+bsa bundles when used outside MO. This IS the case the mod author has tested for and wrote his installation instructions for.

[TheBloke]

Tannin,

Thanks for the further explanations. I understand that filesystem-based links of any kind are out of the question.

But the system that I described does not, I believe, require them. Of course I may be wrong; not knowing the full details of MO's internals.

 

I'd like to restate it again, in light of your further comments.  I've also summarised the intended results in a TLDR section at the end.  Apologies for the length, I do like to go into detail - especially in this case where I want to state my assumptions and thoughts on implementation, in case I am wrong in my expectation of how MO works and how it could be changed.

 

Despite the length of the below, my intention and hope is that the system I describe is rather simple - certainly I feel it would be for the user to use, and more transparent and understandable than how things are now.  And I feel that the internal implementation work in MO should not be too extensive, either, because it just boils down to "autodetect a bunch of specific file names in Data; based on these, generate some specially-flagged mod entries (left pane); these mod entries don't have a filesystem based Filetree, but rather a hardcoded list of files to use when processing that 'mod'"

 

For simplicity, the following example assumes a new MO install, on a vanilla Skyrim Data directory.  The 'upgrade' procedure on an existing install shouldn't be much different.

 

Not covered: advanced options to allow for advanced users to disable/change the proposed default system.  I am assuming that if a system like this was added, a couple of config options would also be added to disable it or change its operation.  

 

 

1. MO scans the Data directory, and detects what ESMs and BSAs exist.  It is looking for certain, named files.  It finds:

• Skyrim.esm, update.esm, Skyrim - *.bsa
• Dawnguard.esm, Dawnguard.bsa
• Hearthfire.esm, Hearthfire.bsa

 

2. Based on finding these known files, it decides that this user has the following vanilla/DLC 'mods':

• Skyrim
• Dawnguard
• Hearthfire

 

3. This new user's mod list (left pane) will not start empty (as it would in current MO), but will contain three default entries: Skyrim, Dawnguard, Hearthfire

 

These default entries are highlighted in bold, to differentiate them from normal mods that the user will install later.  

 

4. Internally, these are special 'vanilla' mods.  MO has created them automatically, and it has flagged them internally to indicate they are vanilla/DLC content.

 

5. There are two types of these 'vanilla' mods, and the type is applied internally by MO to provide the following special UI behaviour:

• Neither type of vanilla 'mod' can be removed from the list or re/uninstalled:
  • right click -> Remove and Uninstall and Reinstall are greyed out for these mods.
• Vanilla type
  • The mod called 'Skyrim'.
  • This has a hardcoded priority 0, and no enable checkbox.
  • Thus it cannot be de-activated, and it cannot be moved from the top of the mod list.
• DLC type
  • The mods called 'Dawnguard', 'Hearthfire', 'Dragonborn' (In this example, the user does not have DB).
  • These can be de-activated (unchecked), and they can be re-prioritised/moved

6. These 'vanilla' mods have hardcoded file lists.  These file lists are set by MO and cannot be changed, and they match the list of known files that MO detected in step 1.

• So the 'Skyrim' mod is hardcoded to contain:
  • Data/Skyrim.esm, Data/Update.esm, and all the Data/Skyrim - *.bsa files.
• The Dawnguard mod is hardcoded to contain:
  • Data/Dawnguard.esm and Data/Dawnguard.bsa

etc.

 

When I say hardcoded - what I mean is, when MO processes this mod, it detects it's a special mod (according to some kind of internal flag indicating as such), and then it reads an internal table to find out 'what files does this special vanilla/DLC mod contain?'   Its Filetree is then provided according to the internal file lookup table I described in step 1.  

 

E.g. whenever MO processes Dawnguard, it knows that the Filetree for this mod is Data/Dawnguard.esm and Data/Dawnguard.bsa.  That is exactly what the user would see if he looked at the UI Filetree for the 'Dawnguard' mod, and (most importantly), that is what files would be used when the virtual Data generator processes this mod.

 

Ideally, this 'hardcoded file list' would actually be a text configuration file, rather than being actually hardcoded.

 

7.  So no changes are made to the Data directory.  No filesystem links are created.  The Filetree of the vanilla/DLC mods exists only by virtue of MO's hardcoded rule for these special vanilla mods, which tells it the files to use (as per point 6) and that these files will be found in Data.

 

-------------

 

 

From that point onwards, everything happens exactly as now in MO.  

 

So what has been achieved?  Internally, not much has changed:- Data is not changed; there are no filesystem links; all the same ESMs and BSAs etc exist in Data as they do now, unchanged.

 

What has changed is that Skyrim, and the official DLCs, have been forced into the standard MO mod structure.  They are now visible in the left pane as usual.  They now obey left-pane ordering like any other mod.  

 

In other words, we've gone from the current system - where MO has a special rule set for vanilla files it finds in Data - to a system where MO still has a special list of vanilla Data files, but it uses this to create standard mod file entries.   Then it processes those Mod file entries exactly like for any other mod.  

 

Because of all this, the user can now re-prioritise the official DLC mods, and in an obvious and familiar and completely transparent way.  Therefore he/she can now interleave the unofficial patches with the official content, to achieve the correct ordering regardless of whether BSAs or loose files are used.

 

So the user can - and will, once he follows a STEP guide or any MO instructions - end up with the following left-pane mod list:  

 

(priority | mod name)

 

0 | Skyrim

1 | Unofficial Skyrim Patch

2 | Hearthfire

3 | Unofficial Hearthfire Patch

4 | Dawnguard

5 | Unofficial Dawnguard Patch

6 | .. other mods

 

I have highlighted in bold the vanilla/DLC mods, that MO added automatically for this user.  I would suggest that bold highlighting would also be applied in the real MO left pane, to indicate that these are 'vanilla game mods', which the user did not install him/herself but which MO automatically created out of his vanilla content.

 

Finally, it provides a new feature for MO - easy, one-click enabling/disabling of DLCs, per-profile.  The user can create a profile that does not use Dawnguard or Hearthfire or Dragonborn, or any combination, if for some reason he prefers to play the game without that DLC.  Perhaps he dislikes something it changes, or perhaps it breaks a mod he really wants to play.  Or, more commonly - perhaps she is a mod creator, who needs to test her mod both with and without different combinations of DLCs.  With this new system, the modder can see that she can now very easily disable any/all DLCs on a per-profile basis, just like she would disable any other mod.

 

TLDR:

 

This is mostly a UI change.  We have forced the vanilla/official content into the standard MO mod structure.  We have replaced MO's hardcoded rules for processing vanilla/DLC content in Data, for a set of hardcoded rules for converting that vanilla/DLC Data content into a list of autodetected, auto-added, 'special' MO mods.

 

When launching the game, vanilla/DLC content is no longer prioritised according to different, behind-the-scenes ordering rules.  Instead, 100% of data for the game - vanilla, DLC, and all mods - is now visible in the left pane, and is prioritised according to the priorities set in the left pane.  

 

The loading of all content, and its ordering, is therefore much more transparent.  It's simpler, and very easy for everyone to understand.   Instructions for most users will be just : "Ensure your content in the left pane matches the order of the STEP guide, then launch BOSS/LOOT (possibly after adding custom BUM rules) to sort the right pane."

 

And the work involved in changing MO would, I am assuming, be fairly minimal:  

• new routine to autodetect content in Data and create left-pane mod list entries
• a new flag to allow 'special/vanilla/DLC' mods that can't be removed/uninstalled, and specifying that the Skyrim mod can't be re-prioritised or de-activated
• special handling for the the Filetree of these vanilla mods, such that their Filetree is not read from "Mod Organizer/mods/<mod>" as usual, but is set by a hardcoded, internal list of specific file names.  Applying both in UI (Filetree tab) and, most importantly, in virtual Data directory generation.
• (ideally) a couple of new advanced configuration options, to disable or modify the default behaviour I describe.  

Edited May 27, 2014 by TheBloke