The new mod order in 1.2.5

[Octopuss]

The following changes should prevent these kinds of errors:

• Move Unofficial Skyrim Patch before Dragonborn
• Move Dawnguard before Dragonborn
• Move Unofficial Dawnguard Patch before Dragonborn
• Move HearthFires before Dragonborn
• Move Unofficial Hearthfire Patch before Dragonborn

 

Shall I? Currently my mod order mostly follows the STEP guide (for the fixes section at the least), and this directly conflicts with it.

I don't really understand the new changes, and the long thread where all of that was discussed was beyond what I could understand.

[blattgeist]

Thanks for the answer DoubleYou and to everyone else for the help so far. Very interesting reads here.. indeed. *taking notes*

 

Regarding updating mods with BSAs. I see no problem there. You installed the mod as a whole (in my case extracted the BSA), a bit later in time there comes a small update. All that needs to be done here is install the update and choose "merge" during the process. The new files will overwrite any old files.

Edited June 25, 2014 by blattgeist

[DoubleYou]

I'm not really sure how to respond to this, especially the part about electron speed..Suffice to say that SSDs perform significantly better than HDDs on every possible metric (besides, perhaps, longevity in write-heavy workloads). In my personal case, CrystalDiskMark shows figures of 350MB/s on sequential reads, and I think around 300MB/s on random 4Kb reads (4KB being significant as it's the filesystem blocksize.) This compares to a maximum of 120MB/s sequential reads I get from my RAID-10 of 4 x 1TB HDDs. Crucially, the random reads on the HDD array is something ridiculous, like 2MB/s.

 

Note also that my SSD isn't even particular good - it's four years old.  Modern ones get 550+ MB/s easily.In more practical terms: moving my Skyrim + 60GB MO mod installation from HDD to SSD reduced my shortcut-to-Skyrim-menu startup time from several minutes, to 30 seconds. That was with all extracted BSAs. Then changing a significant number of my mods from extracted to non-extracted BSAs gave me at most a 1 or 2 second further reduction; 5% maybe. And no in-game performance difference (as measured via FPS and Papyrus script lag time.)If you believe that I could encounter problems that cause no measurable performance change, then I'll need you to elaborate on exactly what problems can be caused and why. 

What do you mean by some BSAs? Which ones are you saying would not be loaded according to install order? (Besides the vanilla/official BSAs of course, the load order of what is what changed in MO 1.2.5.)Could you please point to the specific change(s) in MO that you are referring to when you say this has changed in 1.2.5?Here's the full changelog:

 

Version 1.2.5- mod list context menu split into two menus (one for whole list, one for selected mods) - added option to combine category filters using "or" - added context menu option for deselecting category filters - slightly changed ui on the category filters - simple installer can now be configured to run without any user interaction - extended interface for python plugins - NCC will now report an error if a script tries to extract a non-existent file instead of creating an empty output file - ncc installer plugin now uses a more reliable method to force the installer window to the foreground - missing version on TESV.exe will no longer be reported as an error - number of problems detected by MO is now displayed as a badge on the icon - bsa extraction is now handled in a plugin - added a way for plugins to react to mod installation - re-enabled the automatic fix for asset order problems - added a new mod type that represents files handled externally (i.e. DLCs) as mods in MO - display of "foreign" mods can be limited to only official content - hashes of file names in bsa files are no longer checked all the time - author and description is now read from esp files and displayed in the plugins tab - rewrote the code that fixes modlists after a rename, should be a bit more robust - plugin-list now displays loot messages - loot client now only updates the masterlist once per MO session - new event to notify plugins of changed mod priority - overwrite now shows up in the "checked" category instead of "unchecked" - added pseudo-categories to filter for mo-managed vs. unmanaged mods - deleted mods are now moved to the recycle bin instead of being deleted permanently - added a signal when a plugin is moved - bugfix: the warning dialog upon changing the mod directory didn't have an effect, the process couldn't be canceled - bugfix: conflicts tab in the mod info dialog offered the hide option for files in bsas - bugfix: MO crashed when trying to download via the integrated browser - bugfix: In some cases when a download wasn't started successfully the download urls weren't stored in the meta file so no resume was possible - bugfix: MO tried to resume downloads when it didn't have and download urls - bugfix: downloads couldn't be paused if the download was already broken on the network layer - bugfix: download managear did not recognize a file as downloaded if the download completed before signals were hooked up - bugfix: in-place file replacement was re-broken - bugfix: loot client didn't read list of active mods - bugfix: shortcuts created from MO used the wrong working directory - bugfix: deactivation of bsas didn't stick - bugfix: file hiding mechanism wasn't active - bugfix: executables linked on the toolbar couldn't be removed if the executable was removed first - bugfix: the endorsement-filter couldn't be combined with other filters - bugfix: python interface to repository bridge was broken

 

I will say again that there is no recent change in MO that affects how BSAs are loaded, based on load order. Prior to 1.2.5, MO always resolve conflicts based on the mod list order. If ModA at priority 100 provided FileX as loose, and ModB at priority 110 provided FileX in a BSA, then ModB's file would be the one used in the game. You could see that in both the Conflicts tab, and the Data pane.In fact, Tannin confirmed that during the lengthy Ramifications thread. I think at the start of the thread, some people had the misapprehension that there were some special rules involved, but in fact it's very simple: with regards to mods managed by MO (i.e. that are not just loose in the Data folder), and where the BSAs are ticked in the Data pane, mod list order ruled all, and that is not new.Here are some quotes: 

And here's a nice summary from DoubleYou: 

In other words: MO has always resolved conflicts based purely on mod list order, regardless of whether BSAs or loose files or both are used. This has very much NOT changed in MO 1.2.5!If you are in further doubt, please re-read the Ramifications thread from the start - at the start of the thread, many users had the same misunderstanding that you have now. Tannin and DoubleYou then cleared those up, and I quoted parts of that above.Final point: there is one change in 1.2.5 that changes things with regard to load order: there's a new tickbox on the Archives tab, "Have MO manage archives". If you untick that, then mod order no longer applies; MO then operates like the other mod managers, and loads BSAs according to plugin order.That's new, but that change does not apply by default, nor would most users (and no STEP users) want to enable that change. It would turn MO into a more normal mod manager, where the mod list is mostly irrelevant. We don't want that! Basically, that tickbox turns off the "MO magic", and makes it just load everything according to Plugin order alone. 

Yeah I agree that there's no need to just extract every BSA, for the sake of it - at least not for most users; personally, I do often dig into mod folders in Explorer, and appreciate the ability to do so.Actually there's a further problem that we've not mentioned as yet, which relates to the specific case of installing (large) mods and then applying incremental updates. In the case of always-extracting BSAs, then applying an update that itself contains a BSA, current versions of MO will not properly apply the update. I raised that as a bug, here: Beta 1.2.1: New BSA extraction handling never overwrites existing files, therefore breaks the Merging of patch updates that include BSAs.  

 

That issue can be fixed, and I proposed a solution in that issue report.    But then there's a further issue that potentially can't ever be fixed - it relates to mods that supply different sets of loose files according to fomod installer options, which are duplicates of the same files provided in a BSA.   keithinhanoi detailed that issue, here (scroll down 60% until he quotes me).

 

Therefore I would definitely agree with advice that users should not extract BSAs without prior knowledge, and that they should not extract all BSAs for no reason.  Furthermore, there needs to be specific advice warning users who have extracted BSAs for certain mods that they must not apply incremental updates to them.  This really only applies to very large mods, like Inconsequential NPCs.

 

But nor do I agree with advice to never extract.  Doubly so because it's the exact opposite of the advice in the current STEP guides, and there are number of mods within STEP that require per-file conflict resolution which is impossible if BSAs are not extracted.  

 

Clearly some thought needs to be given to how those STEP guides change, and what advice is given.  But there are definitely a number of mods in the guide which are going to need to include the message "Extract the BSA for this mod, so that you can make the following conflict resolutions..."

I think you totally misunderstood SRB. He was saying that prior to the update MO only managed BSAs in MO, not the ones users may have placed in data directory. With 1.2.5, Mod Organizer can now manage mod BSAs in both MO and data.

[TheBloke]

I think you totally misunderstood SRB. He was saying that prior to the update MO only managed BSAs in MO, not the ones users may have placed in data directory. With 1.2.5, Mod Organizer can now manage mod BSAs in both MO and data.

 

Yeah you may be right - but I had made that exact point earlier: 

BSAs could always overwrote loose files from other mods, in accordance with mod order list, in MO.  This has not changed in MO 1.2.5! What has changed in MO 1.2.5 is that we can now interleave the official vanilla/DLC content, with its unofficial patches.   That was the sole issue that extracted BSAs was causing: because MO always loaded all mod content after the vanilla content, rather than loading BSAs according to Plugin ESM/ESP order like in other managers, it meant that some content from the unofficial patches was not being applied as the designers of the unofficial patches intended.  For example, the Unofficial Patch designers expected that USKP would load before Dragonborn, but in MO, Dragonborn (and DG and HF) would load before any mod was loaded. That is fixed in MO 1.2.5 because now official vanilla/DLC content is treated like a mod.  We can move it in our mod list.  Therefore we can ensure that we have the order:  Skyrim, USKP; Dawnguard, UDGP; HearthFires, UHFP; Dragonborn, UDBP; HiRes DLC, UHRP; other mods...

Then he responded again that that MO 1.2.5 had a change that changed how loose and BSA files were being handled, and he said again that therefore extracting BSAs was never necessary any more.

 

It's that second part that I have been questioning throughout the thread.  SRB has said a couple of times that the changes in MO 1.2.5 mean we shouldn't extract BSAs any more.  But in fact, MO 1.2.5 changed nothing with regard to whether or not we should extract BSAs.  In fact, the changes in MO 1.2.5 were in part to resolve a problem caused by extracting BSAs.

 

In other words, if the advice pre-1.2.5 was "extact BSAs", then MO 1.2.5 has changed nothing about that advice: because in MO 1.2.5, it's safer to extract BSAs than it was before; in particular, with regard to the unofficial patches and their interleaving with official content.

 

I do agree that in fact always extracting BSAs is not a good thing, and that much more care than that should be taken.  But I don't agree that anything has changed in MO 1.2.5 with regard to whether we do or don't extract BSAs.  All that has changed is that, after the long Ramifications thread, we all understand the issue better now.

 

If SRB is referring purely to the handling of non-MO mods, then I apologise for misunderstanding; a lot of what I said in my previous post was therefore unnecessary.  But I still don't understand his original statement:

 

The new install order means there is no need to extract any BSAs.

 

The new install order changes nothing about whether we should extract BSAs; in fact, it means it's now safe to extract Unofficial content BSAs where it was not before.

 

Maybe he means "As a result of the discussions in the Ramifications thread, the current accepted wisdom is that extracting BSAs is not generally a good idea, and we will need to review the STEP guides and find all the places where per-file conflict resolution is being done and decide if those mods should still have BSAs extracted, or if the conflict resolution is not that important."

 

That I could understand totally, and agree with.

Edited June 25, 2014 by TheBloke

[TheBloke]

Thanks for the answer DoubleYou and to everyone else for the help so far. Very interesting reads here.. indeed. *taking notes*

 

Regarding updating mods with BSAs. I see no problem there. You installed the mod as a whole (in my case extracted the BSA), a bit later in time there comes a small update. All that needs to be done here is install the update and choose "merge" during the process. The new files will overwrite any old files.

 

Check out my bug report that I linked above.  It doesn't work like that, at least not any more.

 

Tannin made a change in Beta 1.2.1 that meant that files would not be extracted from a BSA, if an identically named file already existed in the mod folder.  This was done because there were reports of mods that contained identical files both as loose files, and in the BSA.  In that case, the mod author would expect the loose file to always win, because that is Skyrim's default behaviour.  But BSA-extracting users were breaking this expectation, because the BSA was extracted after the loose files were installed, and thus the BSA copy overwrote the local copy.

 

So Tannin changed the BSA extraction method such that it did not extract any file in the BSA if that file already existed locally.  That solves the case of a single mod containing duplicate files, BSA vs loose.

 

But now consider the case where you install a mod and extract the BSA, and later install an update that also includes a BSA:

 

• You install Interesting NPCs.  This includes a few thousand loose files, and a 125MB BSA file containing a few thousand more files.
• You choose to Extract the BSA.  You now have many thousands of loose files, and no BSA - the loose files being the combination of original loose files + extracted-from-BSA loose files.
• All is fine at this point.
• Later, you install an incremental update to Interesting NPCs.  This update includes a few more loose files, and a new copy of the entire BSA.
• You choose Merge in the MO installer.  
• The loose files in the update are written into the mod directory, and correctly overwrite anything already there.
• The BSA is written into the mod directory.
• Now MO asks you if you want to extract the BSA.  Of course, you say yes - if you didn't, that BSA wouldn't be used, because you have all the loose files there from the previous install.
• Now MO extracts the BSA - except, it doesn't extract any file that already exists in the local directory!  And, nearly all files in the BSA do already exist in the local directory; because the BSA contains all the original files, some of which have been updated.  You already extracted that BSA once, in the original mod install, so you already have nearly all of the BSA's contents as loose files.
• The end result is that the only updated files from the BSA that you will get, when you install the update, is any brand new files in that update.  Any changed files (files that were previously in the BSA, and were then changed in the new mod update) will not be unpacked, because MO sees that they already exist.

 

Now that problem is solvable, if Tannin changes how MO handles mod extraction and BSA extraction.  I wrote a proposed feature change in my bug report, that would fix the issue.

 

Unfortunately, there's a second issue, that probably can never be fixed.  keithinhanoi described that, so I'll just quote him:

 

 

This unfortunately creates a major "catch-22" problem when installing updates of BSA + loose file mods which are meant to be installed using the merge option instead of replace (because the update doesn't include the full set of assets.)

 

So, say the author has a "main" BSA with the default assets, and their FOMOD installer includes options with loose files that are intended to override some contained in the BSA. It's normal of the author to expect this to work because without any mod manager or with NMM, the loose files load after all BSA. So I wouldn't blame the author for doing anything wrong by setting things up this way.

 

Now, the latest update of MO deals with this correctly if you want to unpack/extract the mod's BSA to loose files. So, when installing, essentially MO unpacks the BSA first, and then adds the loose files that were installed, overwriting anything unpacked from the BSA with the same name. So that works just as expected.

 

However, if you install an update for the mod, which just includes a replacement BSA, you're going to run into trouble.

 

Because we've previously unpacked the mod's BSA, now everything is mixed together as loose files. How does MO know which ones were originally supplied by the installer as loose files? It doesn't. So when you install this update using the merge option, and allow MO to unpack/extract the BSA, it's actually going to overwrite some of the files that were originally supplied by the previous install as optional loose files.

 

Thinking through this, I can't think of any way of MO knowing what to do automatically, without somehow keeping track of which files were previously supplied as loose files and not in the unpacked BSA.

 

However, there is a bit of a workaround you could use in this case, and it means you need to keep a backup of the previous install archive, or download it again. So when a mod has an update which just contains a replacement for a main BSA that you had previously unpacked, and not any of the optional loose files, you do this:

[*] 

• Re-install the previous version of the mod again, selecting overwrite, but this time don't unpack the BSA.

• Install the update, selecting merge, but also don't unpack the BSA.

• Go to the Assets tab in the right-hand pane, and find the BSA for the mod in question

• Right-click on the BSA, and select Extract...

 

Edited June 25, 2014 by TheBloke

[TheBloke]

This issue of applying updates with BSA extraction enabled is a little convoluted, so I thought I should write a summary:

 

With MO 1.2.5:

 

If you do not extract BSAs:

You have no problem; this issue does not apply to you.

 

If you do extract BSAs:

You must not install any incremental mod update - any install done using MO's "Merge" installation option - if this is a mod that contains BSA files.

 

 

If Tannin implements the proposed feature change detailed in my bug report:

 

If you do not extract BSAs:

You have no problem; this issue does not apply to you.

 

If you do extract BSAs:

• It will be safe to apply an incremental mod update for mods containing BSAs, but only if:
  • The mod contains no fomod installer
  • The mod does contain a fomod installer, but you are certain that this installer does not conditionally install loose files which are also contained within the BSA.
  • Examples which are safe, because they have no installer:  Interesting NPCs, Inconsequential NPCs, Immersive College of Winterhold
  • Examples which are safe, because their installer does not install duplicate loose files:  I need to check this actually, it requires some careful checking of mods!
• It is NOT safe to apply an incremental mod update for mods containing BSAs, which use a fomod installer to apply conditional loose files that are also in the BSA:
  • Example of a mod that does this: Expanded Towns and Cities.
  • There's quite possibly many more examples, too.
• Therefore the safest advice, if Tannin fixes the current issue, will be to only apply incremental updates to mods that have BSAs if they have no installer.

Edited June 25, 2014 by TheBloke

[EssArrBee]

I think you totally misunderstood SRB. He was saying that prior to the update MO only managed BSAs in MO, not the ones users may have placed in data directory. With 1.2.5, Mod Organizer can now manage mod BSAs in both MO and data.

That is exactly what I meant, but I was trying to help people that don't need the rundown of how MO handles stuff. If general users have all these issues it's better to just tell them how to do things without all this extraction and optimization stuff.

 

Also, SSDs do not move data faster, they actually move data much, much slower because of silicon. The difference is that it moves orders of magnitude more data at once than a HHD. Moving data from the magnetic disk to the SATA or whatever connection is measured relative to 'c', while SSDs would be better measured in meters per second. That is what I meant about electrons. I've actually heard Silicon slows down the signal speed to .03c and even your TV's fiber optic cable is running at like 2/3c. SSDs are made with what again?

 

I was also looking ahead to 200+ plugin mod setups with bunches of 2k textures and lots of scripts running, probably higher level characters. The SSD won't save you, only a much better CPU will do that.

[blattgeist]

Oh alright TheBloke.. that is indeed a huge issue. So we need to be aware of mods that bring BSAs and their little updates.. that's why I prefer to install the big packages of mods instead of the little updates. After a while the updates get repackaged into the big file. I had no issues with iNPC and iqNPC updates so far. I regularly work with them.

 

 

If general users have all these issues it's better to just tell them how to do things without all this extraction and optimization stuff.

Tbh. I'm never satisfied with just following orders without thinking. It's better to understand why things happen and how to deal with them.

Edited June 25, 2014 by blattgeist

[DoubleYou]

You had better not be extracting BSAs as a workflow yet...

 

Tannin:

 

groan I found out this is actually two issues: 

 

MO 1.2.5 doesn't allow bsas to overwrite loose files at all.

even after fixing this, bsas from dlcs can't overwrite loose files.

 

 

The first issue will be fixed in 1.2.6, the other will have to wait for 1.2.7

[TheBloke]

You had better not be extracting BSAs as a workflow yet... Tannin:

Thanks for the heads-up!I see Tannin put out 1.2.6 already.

1.2.6 emergency bugfix releasedIt turns out that in my hurry to fix the download issues I didn't test 1.2.5 enough and a major issue crept in that led to file conflicts being resolved differently from how they are supposed to be.1.2.6 only fixes 95% of the problem so you'll have to look out for two things:a) please ensure your DLCs (and other mods not managed by MO, that's the ones without a checkbox) are at the very top of your modlist, only unofficial patches should be up there with them.b) please ensure you didn't unpack the BSAs of the unofficial patches. If you did, please reinstall them and don't unpack this time.

So looks like it's no huge issue, until 1.2.7 we just have to observe the a) and b) rules above.

[DoubleYou]

If you can download 1.2.6....

[TheBloke]

If you can download 1.2.6....

Indeed :) By the time Nexus has updated, he'll have 1.2.7 ready..

[Neovalen]

Well that's inconvenient. Hopefully that gets fixed soonish.

[DoubleYou]

Until then, I recommend using either 1.1.2 or 1.2.4

[TheBloke]

Well that's inconvenient. Hopefully that gets fixed soonish.

1.2.6 seems to fix the most major, unavoidable issues.

 

My understanding of Tannin's changelog is that, while using 1.2.6, we'll be fine as long as we don't extract the Unofficial Patch BSAs, and we have the following priority list at the top of the mod list (first 10 entries, Prio 0 -> Prio 9):

 

 

 

I've asked Tannin for 100% confirmation that that's the right order, and that it's fine to have USKP at the top at Prio 0. But that seems to be what his a) rule in the Changelog requires.

 

As soon as 1.2.6 is available - hopefully not much more than another hour or so - I'll do some tests based on that, confirming that the right files in the right conflict order are getting into the VFS.

Edited June 26, 2014 by TheBloke

[TechAngel85]

Can a staff member update the OP or start a new thread explaining the correct load order? (I'm not 100% sure myself) Once that is done, I'll update 2.2.8 with a notice and a link back to the proper load order. Than will cover us until we get 2.2.9 released.