The new mod order in 1.2.5

[Octopuss]

The following changes should prevent these kinds of errors:

• Move Unofficial Skyrim Patch before Dragonborn
• Move Dawnguard before Dragonborn
• Move Unofficial Dawnguard Patch before Dragonborn
• Move HearthFires before Dragonborn
• Move Unofficial Hearthfire Patch before Dragonborn

 

Shall I? Currently my mod order mostly follows the STEP guide (for the fixes section at the least), and this directly conflicts with it.

I don't really understand the new changes, and the long thread where all of that was discussed was beyond what I could understand.

[TheBloke]

OK regarding the "Priority 0" bug, i.e. where Priority 0 mod seemed to be overwriting all..

I just did a couple of tests in 1.2.6, and  could not re-create it. I don't quite understand the experience of DoubleYou, nor of the guy on the Nexus who had USKP in prio 0 overwriting CCOR (much later). But I can't re-create this in 1.2.6.

Specifically, I:

• Moved the mod "Colored Map Icons", containing the single (loose) file Interface/map.swf, into Priority 0. This file conflicts with SkyUI, which is at Priority 50 or so.
  • Loaded the game, and confirmed I had the default SkyUI map - the file provided by Priority 0 mod had not overwritten the later mod.
• I then made a BSA file of the Colored Map Icons file Interface/map.swf, and put this BSA file into a new mod, which I moved into Priority 0.   (And disabled the Colored Map Icons mod).
  • Again I tested with a loaded game, and confirmed I got the default map.  The BSA file in Priority 0 had not overwritten a later mod.
• Therefore in 1.2.6 I can't re-create the symptom of either a BSA mod or a loose file mod in Priority 0 overwriting a later mod.

 

It might be more complex than I tested, so we'll need to keep an eye on it.  But hopefully Tannin is right and there was no issue, or if there was it's been fixed somehow in 1.2.6 anyway.

[DoubleYou]

Tested and priority 0 bug as I experienced it is entirely eradicated. Also confirmed that BSAs inside data directory are currently not capable of overwriting loose files in Mod Organizer when MO manages BSAs, which seems to be an additional bug found because of the bug.

[TheBloke]

Tested and priority 0 bug as I experienced it is entirely eradicated. Also confirmed that BSAs inside data directory are currently not capable of overwriting loose files in Mod Organizer when MO manages BSAs, which seems to be an additional bug found because of the bug.

Great!

Edited June 26, 2014 by TheBloke

[kylhwch]

Ok, so it sounds like the simplest solution is to sit tight and wait for the 1.27 to be released, at which point we should leave the "MO manages archives" checkbox checked, install the future mods without extracting BSAs, and rely on the mod list to determine overwriting priority. Is that about right?

 

I don't mean to sound snippy. I do appreciate that everyone is working very hard on these complex tools.

[TheBloke]

Ok, so it sounds like the simplest solution is to sit tight and wait for the 1.27 to be released, at which point we should leave the "MO manages archives" checkbox checked, install the future mods without extracting BSAs, and rely on the mod list to determine overwriting priority. Is that about right?

 

I don't mean to sound snippy. I do appreciate that everyone is working very hard on these complex tools.

 

Yup, that'll be simplest.

 

There's no reason to stop extracting BSAs as of 1.2.7, however STEP's advice is apparently changing to be not to do so - presumably so as to bring STEP more in line with what the majority of the modding community do.   There can also be performance benefits in not extracting BSAs, although my experience is that these are likely only noticeable on slow HDDs and not on fast SSDs (though to be fair, with a full STEP installation being in the 10s-of-gigabytes at least, it's quite likely that a large proportion of the userbase are installed on HDD not SSD.)

 

The major downside of not extracting BSAs is that you can't do per-asset conflict resolution.  So you place ModB below ModA in the priority list, therefore for any conflicts, all ModB's assets win.  With a loose-files mod, or an extracted-BSA mod, you would still have the option of choosing some files from ModA to overwrite ModB (by hiding those assets in ModB).   By never extracting BSAs, you lose that ability; so in the case of a situation where you want most of the files from ModB to be priority, but still want some from ModA, then you have no solution any more.  Of course, that's the expected result all the time for 95% of Skyrim mod users, so it can be argued it's far from the end of the world.  (And, unlike with other mod managers, at least with MO you still have the ability to prioritise the mod's files differently to its Plugin(s).)

 

I am putting in a feature request to Tannin which would enable per-file conflict resolution even with unextracted BSAs.  I'm hoping he'll consider worthy of implementing, because that would then remove the final objection to not extracting BSAs.  And STEP could then go back to specifying per-file conflict resolution rules for mods, and still tell people never to extract any BSAs.  (As it now, the STEP guys will have to go through the 2.2.9 list and find any mods which have per-asset resolution rules, and remove those rules if the mods use BSAs.   Unless they decide to suggest extracting the BSAs for those particular mods alone, which is what I would recommend.  Fortunately most such rules usually apply to texture mods, which are often not provided as BSAs.  But I think there will be some cases where per-asset rules will have to be removed, at least if/until Tannin implements per-asset resolution with BSAs.)

Edited June 26, 2014 by TheBloke

[Nozzer66]

Cause I'm in the middle of a complete re-install, I'm still staying with 1.2.6 and downloading mods, dropping them in the suggested STEP order for now and just jumping into game every so often to make sure I haven't made a SNAFU and caused my own CTD Hell.

 

When 1.2.7 comes out I'll worry more about what goes where, etc. Thought I may as well use my time for something worthwhile rather than just sit around doing nothing. :)

[TechAngel85]

Yup, that'll be simplest.

 

There's no reason to stop extracting BSAs as of 1.2.7, however STEP's advice is apparently changing to be not to do so - presumably so as to bring STEP more in line with what the majority of the modding community do.   There can also be performance benefits in not extracting BSAs, although my experience is that these are likely only noticeable on slow HDDs and not on fast SSDs (though to be fair, with a full STEP installation being in the 10s-of-gigabytes at least, it's quite likely that a large proportion of the userbase are installed on HDD not SSD.)

 

The major downside of not extracting BSAs is that you can't do per-asset conflict resolution.  So you place ModB below ModA in the priority list, therefore for any conflicts, all ModB's assets win.  With a loose-files mod, or an extracted-BSA mod, you would still have the option of choosing some files from ModA to overwrite ModB (by hiding those assets in ModB).   By never extracting BSAs, you lose that ability; so in the case of a situation where you want most of the files from ModB to be priority, but still want some from ModA, then you have no solution any more.  Of course, that's the expected result all the time for 95% of Skyrim mod users, so it can be argued it's far from the end of the world.  (And, unlike with other mod managers, at least with MO you still have the ability to prioritise the mod's files differently to its Plugin(s).)

 

I am putting in a feature request to Tannin which would enable per-file conflict resolution even with unextracted BSAs.  I'm hoping he'll consider worthy of implementing, because that would then remove the final objection to not extracting BSAs.  And STEP could then go back to specifying per-file conflict resolution rules for mods, and still tell people never to extract any BSAs.  (As it now, the STEP guys will have to go through the 2.2.9 list and find any mods which have per-asset resolution rules, and remove those rules if the mods use BSAs.   Unless they decide to suggest extracting the BSAs for those particular mods alone, which is what I would recommend.  Fortunately most such rules usually apply to texture mods, which are often not provided as BSAs.  But I think there will be some cases where per-asset rules will have to be removed, at least if/until Tannin implements per-asset resolution with BSAs.)

I currently run everything from a WD Black so I'm on a HDD.

 

As for STEP, the BSA non-extraction is a non-issue at the moment which is why we're not worried about recommending user to not extract their BSAs for this point on. There are no mods in STEP which we have specific conflict resolution instructions for (aka hiding files) that are in BSAs. All mods that we have these instructions for are available as loose files for downloading. That check was done about a month ago so STEP is good to go.

[TheBloke]

 

As for STEP, the BSA non-extraction is a non-issue at the moment which is why we're not worried about recommending user to not extract their BSAs for this point on. There are no mods in STEP which we have specific conflict resolution instructions for (aka hiding files) that are in BSAs. All mods that we have these instructions for are available as loose files for downloading. That check was done about a month ago so STEP is good to go.

 

Ahh, fair enough. That's easy to deal with then.

 

What will be your upgrade advice, for users who have existing STEP installations and thus followed the previous suggestion to extract all BSAs?   Are you going to suggest that they re-install all STEP mods that have BSAs to get everything unextracted?  Or even require it?

 

I think there's no pressing need for them to do that, but I could understand if it was felt important from a simplicity-of-support perspective (to have everyone on a known config.)    If that's not considered a huge issue, then personally I'd recommend requiring the re-installation of the Unofficial Patches, but not make it a requirement to re-install all others just to re-BSA them.  With BSA Extraction disabled, any future update of existing mods will re-BSA them anyway.

[blattgeist]

Even if MO 1.2.7 eradicates the need to extract BSAs for simple conflict resolution it still is handy for people who want to optimize their whole mod organizer directory or edit individual folders. So I'd say there is still some use for it and I hope that Tannin will keep the feature inside MO, although not automatically enabled. Because hat was certainly causing issues for the not so advanced mod users.

Edited June 27, 2014 by blattgeist

[TheBloke]

Even if MO 1.2.7 eradicates the need to extract BSAs for simple conflict resolution it still is handy for people who want to optimize their whole mod organizer directory or edit individual folders.

To be clear, MO 1.2.5 and 1.2.7 changes nothing regarding the need (or lack of) to extract BSAs for simple conflict resolution.  The decision as to whether to extract or not is exactly the same as it was in 1.2.1 and before.  In fact, the changes in MO 1.2.5 made it safer to extract BSAs - at least with regard to the Unofficial Patches.  So if anything, one could argue that the core changes in MO 1.2.5 and later encourage BSA extraction :)   However BSA extraction is not in fact encouraged, certainly not by Tannin and not by STEP now either.  And we all learnt a lot about it during the lengthy Ramifications thread.  The main outstanding issue with it, as we've discussed before, is that it breaks the ability to Merge update mods. Incidentally, I thought of another broken case with merging updates when extracting all BSAs: a situation where a new update includes the BSA file, but some files have been removed from the BSA compared to the original release.  Users who extract all BSAs will not get those files removed, ending in a situation where, after merging in an update, they still have files that the mod author intended to be deleted in the latest release.  At best, that means they have unnecessary clutter; at worst, the mod will malfunction because of the extra files - that could particularly be an issue in the case of scripts. This is another issue that can never be solved, and I think basically means that users who extract BSAs must never ever apply any incremental update, via the Merge installation feature, for any mod that uses BSAs. 

So I'd say there is still some use for it and I hope that Tannin will keep the feature inside MO, although not automatically enabled. Because hat was certainly causing issues for the not so advanced mod users.

I'm no longer worried that Tannin plans to remove it completely.  It's known that he doesn't like the feature much, but despite that he still implemented in 1.2.5 the Non-MO mod handling which improves compatibility when extracting BSAs - at least in the case of the Unofficials - and wasn't really that important for users who don't extract.   He did also move BSA extraction to an external, default-disabled plugin, to discourage its use.  But I think that's as far as he plans to go.  Default disabled, discouraged in documentation and discussion, but remaining available. In fact, now that it's a plugin, we can't ever really lose the feature - even if he removed the plugin from future releases, we could always copy it back in from the current release (so long as the plugin API doesn't change hugely in the future.)  But I don't think he'll ever remove it completely. Plus of course there's the Archives tab right-click -> Extract BSA feature as well.  Not quite as convenient, because you need to browse to the mod directory and it doesn't delete the BSA afterwards, but does provide another way to extract BSAs, and one that works for installed mods rather than requiring a re-install. Edited June 27, 2014 by TheBloke

[blattgeist]

 

This is another issue that can never be solved, and I think basically means that users who extract BSAs must never ever apply any incremental update, via the Merge installation feature, for any mod that uses BSAs.

I can only imagine 2 workarounds for this:

1. The update itself contains no BSA but is instead provided as loose files.

2. Users will have to install the update (which comes with a BSA) in a separate folder and call it for example "iqNPC update 2.01". After that they have to go into the explorer and manually copy over the contents from the now extracted update into the original mod folder (i.ex. iqNPC 2.0) folder and overwrite everything.

 

By the way, do I need to hit the "back-date BSAs" button in MO if I have Skyrim-Invalidation.bsa active? I have never done that so far. Does the load order of skyrim-invalidation.bsa play a role? I can move it around...

 

I also have no '"bInvalidateOlderFiles" lines in my skyrim inis, if that plays a role. If all that is needed, to apply invalidation of older files, is to have the BSA around, then I'm fine with not hitting the button. But I need some insight here :)

Edited June 27, 2014 by blattgeist

[TheBloke]

I can only imagine 2 workarounds for this:1. The update itself contains no BSA but is instead provided as loose files.2. Users will have to install the update (which comes with a BSA) in a separate folder and call it for example "iqNPC update 2.01". After that they have to go into the explorer and manually copy over the contents from the now extraded update into the original mod folder (i.ex. iqNPC 2.0) folder and overwrite everything.

 1, yeah definitely.  2, that solves the general issue that we currently have, where we can't ever merge in a BSA mod if we extracted the original BSA.  That issue exists because as of MO 1.2.1, MO won't extract from a BSA if the local file already exists. So if you extracted the original BSA, then nearly all of the files in the updated BSA will already exist, and MO won't extract them.  Hence any files that were changed in the update will not be updated.  The solution you described fixes that manually, and I proposed an automated version of that solution to Tannin (bug tracker link.) But I've now realised that there's little point him implementing that, because merge updates can never be completely safe.  The reason is the new issue I described in my last post -  where the new update BSA has removed files, compared to the previous copy of that BSA.  The solution you described doesn't help with that.  If the original mod had FileA in its BSA, but the new BSA had removed that file, then however you apply the update BSA, you will still have a copy of FileA remaining from the original install and extract.  You would have to manually compare the contents of the original BSA and the new one, to work out which files you must now delete.   So the only solution is as you said in 1: if you extract, only use merge update when no BSA is involved.

Edited June 27, 2014 by TheBloke

[blattgeist]

 

The solution you gave doesn't help with that.  If r original mod had FileA in its BSA, but the new BSA had removed that, then however you apply the update BSA, you will still have a copy of FileA remainkng from the original install and extract.  You would have to manually compare the contents of both the original BSA and the new one, to work out which files you must now delete.

True, if files are left behind that's a big problem. But I think that a mod author should be aware of that issue and give proper instructions about which files to remove when updatting. Because as far as I know no mod managing tool up to date can handle things like this. Either the mod author provides a complete package with the removed files not included or they give proper instructions about what to do when updating.

 

Btw. I edited my post above with a question.

[TheBloke]

True, if files are left behind that's a big problem. But I think that a mod author should be aware of that issue and give proper instructions about which files to remove when updatting. Because as far as I know no mod managing tool up to date can handle things like this. Either the mod author provides a complete package with the removed files not included or they give proper instructions about what to do when updating.

But the point is, this is only an issue for users who extract BSAs. And no users in general extract BSAs, besides a few who use MO.What I described above is only an issue when the mod author supplies a BSA in his original mod, then applies an incremental update with a new BSA. If that new BSA has removed some files inside it, then a BSA-extracting user will not have those removed when he applies the update via a Merge. Every other user has no issue, because he didn't extract the BSA, and therefore the incremental update just replaced the old BSA with the new one.

By the way, do I need to hit the "back-date BSAs" button in MO if I have Skyrim-Invalidation.bsa active? I have never done that so far. Does the load order of skyrim-invalidation.bsa play a role? I can move it around... I also have no '"bInvalidateOlderFiles" lines in my skyrim inis, if that plays a role. If all that is needed, to apply invalidation of older files, is to have the BSA around, then I'm fine with not hitting the button. But I need some insight here :)

Regarding invalidation - I'm not really sure. I don't have Skyrim - Invalidation.bsa, and I haven't hit the profile option "Automatically Invalidate Archives".Maybe I'm doing something wrong? But I thought I read that archive invalidation was no longer an issue really in Skyrim? That's what I think I read when I googled it, after first seeing that option in MO.I only play Skyrim, not any other games supported by MO. And I think I also read that, whatever issues exist in general, MO handles automatically? I haven't ticked the "Automatic Invalidation" option in my profile, because I thought that didn't apply to Skyrim.I didn't even realise until recently that there was a "Skyrim - Invalidation.bsa". So I don't really know. A user mentioned that file recently, I think in this post or another one on this forum, and then came back "Ah MO handles that automatically, no problem." Which was my understanding too.I've never really understood the invalidation issue, so someone else will need to comment in more detail - but my understanding that, at least when using MO, there was nothing to worry about? (Bearing in mind that MO seems to create dummy BSAs every time you launch Skyrim or any executable - it maps all your archives to archives called aaa, aab, aac, etc; at least that's what the logs indicate. So I would have assumed that the dates could be set appropriately automatically at the same time. But I'm not completely sure.) Edited June 27, 2014 by TheBloke

[TheBloke]

Right I just did some more research, and can quote Tannin himself (from 2012!) 

@doveman:Skyrim - Invalidation.bsa is not part of the game, it's part of MOs "Automatic Archive Invalidation"-Feature. Afaik it should never hurt to enable it.f you go to settings->workarounds and click "Backdate BSA files" you don't need it though. If you haven't backdated your bsas, you might find some textures/meshes from mods don't work without the invalidation.bsa.

I had pressed Backdate BSA files in the past, and thought I was then good to go. But now I'm wondering if it's something I have to do each time BSAs change? Because it seems to perform some action, likely physically on the filesystem. Suggesting it needs to be repeated as mods change?So yeah, maybe we should either be clicking that button regularly (whenever BSAs from mods change), or we should be be enabling Automatic Invalidation in the profiles and putting Invalidation.BSA in the load order. Definitely not neither of those, though both is not necessary.Personally I now plan to do the former, and just press that button every time I change mods. I'd rather do that than have an extra mod list entry.So clearly I must have been doing something wrong up until now, because it's been a while since I hit the Backdate BSAs button. Not that I can say I've ever noticed any problems, though I've only recently started having a lot of BSAs, before that I extracted everything except vanilla. If the main symptom of problems is just some missing textures, I may have had that and just not noticed. Edited June 27, 2014 by TheBloke