The new mod order in 1.2.5

[Octopuss]

The following changes should prevent these kinds of errors:

• Move Unofficial Skyrim Patch before Dragonborn
• Move Dawnguard before Dragonborn
• Move Unofficial Dawnguard Patch before Dragonborn
• Move HearthFires before Dragonborn
• Move Unofficial Hearthfire Patch before Dragonborn

 

Shall I? Currently my mod order mostly follows the STEP guide (for the fixes section at the least), and this directly conflicts with it.

I don't really understand the new changes, and the long thread where all of that was discussed was beyond what I could understand.

[TheBloke]

Wait there is?Do we make a dummy mod and put it there?

 

Or just a mod that has no conflicts.  I have Appropriately Attired Jarls in position 0. It has no conflicts, so no problems.

 

The new install order means there is no need to extract any BSAs.

 

Install order is all that matters now. Extraction and all that stuff is irrelevant now.

 

Can you elaborate?  Do you mean the new install order in MO 1.2.5?  Or has STEP been updated?

 

Because previously STEP had some examples where it said "Only use the following files from mod X", or "Hide the following files from mod Y".  Maybe all such mods are provided as loose files?  But if they're provided as BSAs, then this kind of per-file conflict resolution is not possible unless you extract those BSAs.  

 

You cannot currently hide individual files if they are provided within a BSA - though I have discussed with Tannin asking for that ability to be added, which I will be putting into an official Feature Request shortly (today or tomorrow.)

 

I've moved to not extracting all BSAs, and if I never have to do per-file conflict resolution on a mod I'll likely leave its BSA unextracted.  But I do have several mods where I've done per-file conflict fixing - e.g. Mod A is before Mod B in the mod list, but I want one or two files from Mod A to overwrite Mod B.

 

Note I've not checked the STEP guide thoroughly to remind myself on which mods it says "only use these files" or "hide these files", so maybe it does not say that for any mod that provides BSAs?

 

But my point is, I did not think that it could just be said "never extract any BSAs any more", because there may be some cases where per-file conflict resolution is necessary or desired?

Edited June 25, 2014 by TheBloke

[Octopuss]

So how exactly the bug works? If I put a mod with exactly one texture in it on position 0, will this particular texture take priority over any other mod placed later which replaces the exactly same texture, or is it more complicated than that?

[blattgeist]

So how exactly the bug works? If I put a mod with exactly one texture in it on position 0, will this particular texture take priority over any other mod placed later which replaces the exactly same texture, or is it more complicated than that?

 

It's as you described Octopuss.

 

Install order is all that matters now. Extraction and all that stuff is irrelevant now.

To me it still matters for example when running SMCO over individual textures. That's why I thought I'd get a proper answer: A simple yes or no to my question from before.

Edited June 25, 2014 by blattgeist

[EssArrBee]

SMCO the optimizer? You'd do that for the individual folder instead of running it through MO and doing the optimization, so it wouldn't matter.

 

The performance increase from running textures in archives would be better for the CPU, but once unpacked unoptimized textures are worse for GPU. You could unpack, optimize, and then repack using archive.exe that comes with the creation kit. It's in your root Skyrim folder.

 

You don't want to go optimizing stuff in the patches anyways and why would you? Those textures have had special fixes applied to them and you could an optimizer could undo some of them. Leave the patches in their BSAs as they are designed to packed up and loaded early.

[TheBloke]

So how exactly the bug works? If I put a mod with exactly one texture in it on position 0, will this particular texture take priority over any other mod placed later which replaces the exactly same texture, or is it more complicated than that?

 

Not more complicated than that. no.

 

Easiest way to think about it:  Priority 0 is being treated like priority 999,999.  It's like it's below every other mod, and so it wins any conflicts it has with other mods.  If it has no conflicts, then it doesn't matter.

Edited June 25, 2014 by TheBloke

[blattgeist]

Hm.. fair enough. That means.. to untick BSA extractor when installing USPs. You got a point with optimizing USP textures. And I always run SMCO over my whole mod folder, because it's easier. I've had no issues so far.

Edited June 25, 2014 by blattgeist

[TheBloke]

Hm.. fair enough. That means.. to untick BSA extractor when installing USPs. You got a point with optimizing USP textures. And I always run SMCO over my whole mod folder, because it's easier. I've had no issues so far.

Yes there's definitely no need to unpack the unofficial patches.

 

Though the changes added by 1.2.5, allowing the interleaving of official/DLC content with unofficial patches, means that there's also now no huge reason not to unpack them.  Before, it caused a bug.  That is now fixed in 1.2.5 because we can interleave the official content/DLC with its unofficial patch.

 

However some users do find better peformance when not extracting BSAs.  I have not noticed much difference, when running the game from a fast SSD.  Running it from slow HDDs would likely show a big performance improvement if you use all BSAs instead of all extracted.

Edited June 25, 2014 by TheBloke

[EssArrBee]

Hm.. fair enough. That means.. to untick BSA extractor when installing USPs. You got a point with optimizing USP textures. And I always run SMCO over my whole mod folder, because it's easier. I've had no issues so far.

You shouldn't be extracting mods default. It is bad for performance either way. Loose files take to long to retrieve from disk. Most the mods that are packed as BSAs should be left that way. That is why we had the discussions about it the last month or so. MO allows the BSAs to overwrite loose files so install order rules over all now. Also, don't optimize Cabal's and Vurt's textures, optimizers do some weird stuff to their textures.

[TheBloke]

You shouldn't be extracting mods default. It is bad for performance either way. Loose files take to long to retrieve from disk. Most the mods that are packed as BSAs should be left that way. That is why we had the discussions about it the last month or so. MO allows the BSAs to overwrite loose files so install order rules over all now. Also, don't optimize Cabal's and Vurt's textures, optimizers do some weird stuff to their textures.

 

I've benchmarked and found no noticeable performance delta between having all extracted BSAs and not extracted BSAs, when using fast SSD storage for my Skyrim + mod installation.   I agree that on HDDs, the performance benefits may be worthwhile.  On SSDs, file fragmentation is not an issue, and random reads are almost as fast as sequential reads, where on HDDs random reads are orders-of-magnitude slower than sequential.  Hence on HDDs, packing all your files together in contiguous archives can be a big win.

 

The downside of not extracting BSAs is the inability to resolve conflicts on a per-file basis, and the inability to easily browse and read files within a mod (e.g. through Explorer.)  The latter won't be a disadvantage for most people, just maybe modders.  The former though can be a bigger issue, at least for advanced users.

 

Regarding "MO allows the BSAs to overwrite loose files so install order rules over all now." - I think you are misunderstanding the changes that happened in 1.2.5

 

BSAs could always overwrote loose files from other mods, in accordance with mod order list, in MO.  This has not changed in MO 1.2.5!

 

What has changed in MO 1.2.5 is that we can now interleave the official vanilla/DLC content, with its unofficial patches.   That was the sole issue that extracted BSAs was causing: because MO always loaded all mod content after the vanilla content, rather than loading BSAs according to Plugin ESM/ESP order like in other managers, it meant that some content from the unofficial patches was not being applied as the designers of the unofficial patches intended.  For example, the Unofficial Patch designers expected that USKP would load before Dragonborn, but in MO, Dragonborn (and DG and HF) would load before any mod was loaded.

 

That is fixed in MO 1.2.5 because now official vanilla/DLC content is treated like a mod.  We can move it in our mod list.  Therefore we can ensure that we have the order:  Skyrim, USKP; Dawnguard, UDGP; HearthFires, UHFP; Dragonborn, UDBP; HiRes DLC, UHRP; other mods...

 

That is why I asked you earlier why you were saying that one should never extract any BSAs, when in fact the STEP guide features some examples of per-file conflict resolution, which can only be done if you extract the BSAs for those mods (unless they are mods that don't use BSAs.)   Nothing has changed in MO 1.2.5 that affects that.   If you want to put Mod B below Mod A in your mod order, but have a couple of files from Mod A take precedence over Mod B, you still need extracted BSAs.  There is no option to "Hide files" if those files are contained within BSAs.

 

At least there isn't at the moment; as I mentioned earlier, I am working on a feature proposal for Tannin which, if implemented, would allow us to do per-file conflict resolution even with mods that provide all files as BSAs.  If that is implemented, then your statement "never extract BSAs" might be correct for all users.  At the moment, it is not.  (Though I agree that, by default, users should not extract BSAs, and there is no reason to extract a lot of BSAs from a lot of mods.  It's only when you want to do more finely grained file conflict management.)

Edited June 25, 2014 by TheBloke

[blattgeist]

You shouldn't be extracting mods default. It is bad for performance either way. Loose files take to long to retrieve from disk. Most the mods that are packed as BSAs should be left that way. That is why we had the discussions about it the last month or so. MO allows the BSAs to overwrite loose files so install order rules over all now. Also, don't optimize Cabal's and Vurt's textures, optimizers do some weird stuff to their textures.

Letting mod files unextracted means that I lose a huge deal of conflict resolution ability. I need to manually hide files when managing my load order. So that's no option. And I don't want to manually extract and repack archives everytime I install a mod or an update. Here is my new archive order: https://s7.directupload.net/images/140625/qtqakglg.jpg

 

 

By the way, very good explanation @ TheBloke. Thanks for that.

 

 

The sole reason why I accepted not to extract the USP BSAs as suggested here is because of the argument that was brought up by EssArrBee to avoid optimizing USP and UHRP textures. SMCO by default does not optimize archives unless you tell it to do so.

 

 

 (Though I agree that, by default, users should not extract BSAs, and there is no reason to extract a lot of BSAs from a lot of mods.  It's only when you want to do more finely grained file conflict management.)

I consider myself as very picky when it comes to load- and install order optimization. I document every little bit that helps to understand how particular mods should be sorted. So conflict resolution between separate files is a must have for me. It's also nice to check into mod folders outside of MO and do stuff like editing nifs, dds files and look into scripts.

Edited June 25, 2014 by blattgeist

[TheBloke]

Letting mod files unextracted means that I lose a huge deal of conflict resolution ability. I need to manually hide files when managing my load order. So that's no option. And I don't want to manually extract and repack archives everytime I install a mod or an update. Here is my new archive order: https://s7.directupload.net/images/140625/qtqakglg.jpg

Yeah that order looks fine with regard to the official content and unofficial patches.  Well, I don't know about Director's Cut, I guess that's a German only thing?  But I'm assuming you know that you need to put that before the between DB and the HiRs DLCs?  It's treated like official content?

 

 

Note that I still wouldn't recommend extracting the BSAs for the Unofficial Patches - simply because there's no reason or benefit in doing so. You have them nearly at the top of your load order, they will be overwritten by all other mods anyway.

 

 

 

Here's mine, FYI:  

Edited June 25, 2014 by TheBloke

[EssArrBee]

I've benchmarked and found no noticeable performance delta between having all extracted BSAs and not extracted BSAs, when using fast SSD storage for my Skyrim + mod installation.   I agree that on HDDs, the performance benefits may be worthwhile.  On SSDs, file fragmentation is not an issue, and random reads are almost as fast as sequential reads, where on HDDs random reads are orders-of-magnitude slower than sequential.  Hence on HDDs, packing all your files together in contiguous archives can be a big win.

 

The downside of not extracting BSAs is the inability to resolve conflicts on a per-file basis, and the inability to easily browse and read files within a mod (e.g. through Explorer.)  The latter won't be a disadvantage for most people, just maybe modders.  The former though can be a bigger issue, at least for advanced users.

 

Regarding "MO allows the BSAs to overwrite loose files so install order rules over all now." - I think you are misunderstanding the changes that happened in 1.2.5

 

BSAs could always overwrote loose files from other mods, in accordance with mod order list, in MO.  This has not changed in MO 1.2.5!

 

What has changed in MO 1.2.5 is that we can now interleave the official vanilla/DLC content, with its unofficial patches.   That was the sole issue that extracted BSAs was causing: because MO always loaded all mod content after the vanilla content, rather than loading BSAs according to Plugin ESM/ESP order like in other managers, it meant that some content from the unofficial patches was not being applied as the designers of the unofficial patches intended.  For example, the Unofficial Patch designers expected that USKP would load before Dragonborn, but in MO, Dragonborn (and DG and HF) would load before any mod was loaded.

SSDs close the gap, but you are still going with an unoptimized method of retrieval with with regards to your CPU. You might not notice an in game difference, but you have a much higher potential to have issues because of the wasted clock cycles your CPU has to wait for retrieval of larger files. Basically, SSDs can't move data faster than than HDDs. Electrons still move at the same speed, it's the amount of electrons that are moving that is greater with SSDs. That's the big advancement, and it's why interface connections are getting so much better since SSDs came around. Have you noticed how fast SATA and PCI standards have been turning over lately? It's really exciting to see it happen.

 

Anyways, yes I knew that MO could make some BSAs load based on install order, but install order did not rule all because MO only managed MO install order. Now it it does rule over all with it's new update. You can extract some BSAs if you need to hide a file here or there, but overall how many is that really? Maybe one or two. Unless that problem exists, leave them alone.

[blattgeist]

 

Yeah that order looks fine with regard to the official content and unofficial patches.  Well, I don't know about Director's Cut, I guess that's a German only thing?  But I'm assuming you know that you need to put that before the between DB and the HiRs DLCs?  It's treated like official content?

The Skyrim Director's Cut is a compilation of all of the german USPs (comes with the 4 different USP plugins ofc). I chose to extract the USP BSAs that came with that compilation and manually hid every folder+Esp (textures, meshes, scripts etc.) in MO except for the Strings folder. The german strings folder that comes with the SDC is important, because it fixes numerous spelling mistakes in the german version. I then dragged the SDC behind the other USPs, so that the official DLCs don't overwrite the german strings of the SDC.

 

It's treated like official content yes, but it needs to load after it in order to take effect (only talking about the strings here).

 

 

 

Note that I still wouldn't recommend extracting the BSAs for the Unofficial Patches - simply because there's no reason or benefit in doing so. You have them nearly at the top of your load order, they will be overwritten by all other mods anyway.

I basically reinstalled the USPs after our talk here, by choosing not to extract BSAs during the installation. Though.. what I really did was delete the already extracted content and replace it with the BSA files from the installation archives.

 

 

Here's mine, FYI:  

Where is your skyrim - invalidation.bsa? Oh.. I just read about it.. it's part of MO's archive system. Nvm then.

 

 

What the heck.. that other screenshot that I posted does not reflect my actual new archive load order... *sigh* reuploading.

Here: https://s1.directupload.net/images/140625/mjtmb8el.jpg

This one shows the USP BSAs with the changes I made.

 

 

 

Oh god.. these fish sticks are so yummy! *totally unrelated note*

Edited June 25, 2014 by blattgeist

[DoubleYou]

Does that mean that the old install order (1. DLCs 2. HighResPacks 3. USPs 4. UHRP) is fine for people that extract the USP BSAs?

No. It is not fine. It is only fine for people who do not allow Mod Organizer to handle BSAs and do not extract BSAs.

[TheBloke]

SSDs close the gap, but you are still going with an unoptimized method of retrieval with with regards to your CPU. You might not notice an in game difference, but you have a much higher potential to have issues because of the wasted clock cycles your CPU has to wait for retrieval of larger files. Basically, SSDs can't move data faster than than HDDs. Electrons still move at the same speed, it's the amount of electrons that are moving that is greater with SSDs. That's the big advancement, and it's why interface connections are getting so much better since SSDs came around. Have you noticed how fast SATA and PCI standards have been turning over lately? It's really exciting to see it happen.

I'm not really sure how to respond to this, especially the part about electron speed..

 

Suffice to say that SSDs perform significantly better than HDDs on every possible metric (besides, perhaps, longevity in write-heavy workloads). In my personal case, CrystalDiskMark shows figures of 350MB/s on sequential reads, and I think around 300MB/s on random 4Kb reads (4KB being significant as it's the filesystem blocksize.) This compares to a maximum of 120MB/s sequential reads I get from my RAID-10 of 4 x 1TB HDDs. Crucially, the random reads on the HDD array is something ridiculous, like 2MB/s.

 

Note also that my SSD isn't even particular good - it's four years old.  Modern ones get 550+ MB/s easily.

 

In more practical terms: moving my Skyrim + 60GB MO mod installation from HDD to SSD reduced my shortcut-to-Skyrim-menu startup time from several minutes, to 30 seconds. That was with all extracted BSAs. Then changing a significant number of my mods from extracted to non-extracted BSAs gave me at most a 1 or 2 second further reduction; 5% maybe. And no in-game performance difference (as measured via FPS and Papyrus script lag time.)

 

If you believe that I could encounter problems that cause no measurable performance change, then I'll need you to elaborate on exactly what problems can be caused and why.

 

Anyways, yes I knew that MO could make some BSAs load based on install order, but install order did not rule all because MO only managed MO install order. Now it it does rule over all with it's new update.

What do you mean by some BSAs? Which ones are you saying would not be loaded according to install order? (Besides the vanilla/official BSAs of course, the load order of what is what changed in MO 1.2.5.)

 

Could you please point to the specific change(s) in MO that you are referring to when you say this has changed in 1.2.5?

 

Here's the full changelog:

 

 

Version 1.2.5- mod list context menu split into two menus (one for whole list, one for selected mods) - added option to combine category filters using "or" - added context menu option for deselecting category filters - slightly changed ui on the category filters - simple installer can now be configured to run without any user interaction - extended interface for python plugins - NCC will now report an error if a script tries to extract a non-existent file instead of creating an empty output file - ncc installer plugin now uses a more reliable method to force the installer window to the foreground - missing version on TESV.exe will no longer be reported as an error - number of problems detected by MO is now displayed as a badge on the icon - bsa extraction is now handled in a plugin - added a way for plugins to react to mod installation - re-enabled the automatic fix for asset order problems - added a new mod type that represents files handled externally (i.e. DLCs) as mods in MO - display of "foreign" mods can be limited to only official content - hashes of file names in bsa files are no longer checked all the time - author and description is now read from esp files and displayed in the plugins tab - rewrote the code that fixes modlists after a rename, should be a bit more robust - plugin-list now displays loot messages - loot client now only updates the masterlist once per MO session - new event to notify plugins of changed mod priority - overwrite now shows up in the "checked" category instead of "unchecked" - added pseudo-categories to filter for mo-managed vs. unmanaged mods - deleted mods are now moved to the recycle bin instead of being deleted permanently - added a signal when a plugin is moved - bugfix: the warning dialog upon changing the mod directory didn't have an effect, the process couldn't be canceled - bugfix: conflicts tab in the mod info dialog offered the hide option for files in bsas - bugfix: MO crashed when trying to download via the integrated browser - bugfix: In some cases when a download wasn't started successfully the download urls weren't stored in the meta file so no resume was possible - bugfix: MO tried to resume downloads when it didn't have and download urls - bugfix: downloads couldn't be paused if the download was already broken on the network layer - bugfix: download managear did not recognize a file as downloaded if the download completed before signals were hooked up - bugfix: in-place file replacement was re-broken - bugfix: loot client didn't read list of active mods - bugfix: shortcuts created from MO used the wrong working directory - bugfix: deactivation of bsas didn't stick - bugfix: file hiding mechanism wasn't active - bugfix: executables linked on the toolbar couldn't be removed if the executable was removed first - bugfix: the endorsement-filter couldn't be combined with other filters - bugfix: python interface to repository bridge was broken

 

 

 

I will say again that there is no recent change in MO that affects how BSAs are loaded, based on load order. Prior to 1.2.5, MO always resolve conflicts based on the mod list order. If ModA at priority 100 provided FileX as loose, and ModB at priority 110 provided FileX in a BSA, then ModB's file would be the one used in the game. You could see that in both the Conflicts tab, and the Data pane.

 

In fact, Tannin confirmed that during the lengthy Ramifications thread. I think at the start of the thread, some people had the misapprehension that there were some special rules involved, but in fact it's very simple: with regards to mods managed by MO (i.e. that are not just loose in the Data folder), and where the BSAs are ticked in the Data pane, mod list order ruled all, and that is not new.

 

Here are some quotes:

 

What I'm saying is:

- If the mod with the bsa          is higher in the installation order than the one with loose files then the bsa          takes precedence

- If the mod with the loose files is higher in the installation order than the one with bsa          then the loose files take precedence

 

Or, to phrase it more plainly (like I did before)

- If a mod has higher installation order then it takes precedence. The format just does not matter

 

I honestly don't understand how MO is complicated when it gets rid of a whole category of complexity...

And here's a nice summary from DoubleYou:

 

Okay, this isn't really as hard as ya'll are making it. I thought the wiki made MO BSA understanding clear, from the Priorities tab:

 

Just to make everything perfectly clear:

[*]If BSA is checked in the Archives tab, forget that the files are inside a BSA. Totally. It is as if it were extracted. The only thing you can't do is hide any files within the BSA. Typical Mod Organizer black magic. Don't worry about the how. Only Tannin and a few of us enlightened ones know.

[*]If an asset is in a BSA and that BSA is checked in the Archives tab and another mod with a lower priority contains a loose version of the same asset, the asset from the BSA in higher priority will indeed be used by Mod Organizer in place of the loose asset.

In other words: MO has always resolved conflicts based purely on mod list order, regardless of whether BSAs or loose files or both are used. This has very much NOT changed in MO 1.2.5!

 

If you are in further doubt, please re-read the Ramifications thread from the start - at the start of the thread, many users had the same misunderstanding that you have now. Tannin and DoubleYou then cleared those up, and I quoted parts of that above.

 

Final point: there is one change in 1.2.5 that changes things with regard to load order: there's a new tickbox on the Archives tab, "Have MO manage archives". If you untick that, then mod order no longer applies; MO then operates like the other mod managers, and loads BSAs according to plugin order.

 

That's new, but that change does not apply by default, nor would most users (and no STEP users) want to enable that change. It would turn MO into a more normal mod manager, where the mod list is mostly irrelevant. We don't want that! Basically, that tickbox turns off the "MO magic", and makes it just load everything according to Plugin order alone.

 

You can extract some BSAs if you need to hide a file here or there, but overall how many is that really? Maybe one or two. Unless that problem exists, leave them alone.

Yeah I agree that there's no need to just extract every BSA, for the sake of it - at least not for most users; personally, I do often dig into mod folders in Explorer, and appreciate the ability to do so.

 

Actually there's a further problem that we've not mentioned as yet, which relates to the specific case of installing (large) mods and then applying incremental updates. In the case of always-extracting BSAs, then applying an update that itself contains a BSA, current versions of MO will not properly apply the update. I raised that as a bug, here: Beta 1.2.1: New BSA extraction handling never overwrites existing files, therefore breaks the Merging of patch updates that include BSAs.  

 

That issue can be fixed, and I proposed a solution in that issue report.    But then there's a further issue that potentially can't ever be fixed - it relates to mods that supply different sets of loose files according to fomod installer options, which are duplicates of the same files provided in a BSA.   keithinhanoi detailed that issue, here (scroll down 60% until he quotes me).

 

Therefore I would definitely agree with advice that users should not extract BSAs without prior knowledge, and that they should not extract all BSAs for no reason.  Furthermore, there needs to be specific advice warning users who have extracted BSAs for certain mods that they must not apply incremental updates to them.  This only applies to very large mods which both use BSAs and provide incremental updates, for example Interesting NPCs.

 

But nor do I agree with advice to never extract.  Doubly so because it's the exact opposite of the advice in the current STEP guides, and there are number of mods within STEP that require per-file conflict resolution which is impossible if BSAs are not extracted.  

 

Clearly some thought needs to be given to how those STEP guides change, and what advice is given.  But there are definitely a number of mods in the guide which are going to need to include the message "Extract the BSA for this mod, so that you can make the following conflict resolutions..."

Edited June 25, 2014 by TheBloke