Improving Conflict Detection by Removing Duplicates

[fireundubh]

I have installed a number of location mods that appear to have conflicting assets.

• Heljarchen Farm
• Immersive College of Winterhold
• Morskom Estate
• Vjarkell Castle
• Windstad Mine

I extracted each BSA and generated MD5 hashes for every file in each directory.

 

I imported the results into a spreadsheet and confirmed that the totals below match Mod Organizer's results.

• Heljarchen Farm has 74 conflicts with the other mods.
• Immersive College of Winterhold has 49 conflicts with the other mods.
• Morskom Estate has 151 conflicts with the other mods.
• Vjarkell Castle has 2 conflicts with the other mods.
• Windstad Mine has 130 conflicts with the other mods.

Mod Organizer finds a total of 406 conflicts between these mods but most of these conflicts aren't meaningful.

 

When I removed the duplicates and filtered for actual conflicts (same file names, same file paths, different MD5 hashes), there was a dramatic difference in the number of conflicts.

• Heljarchen Farm has 6 actual conflicts with the other mods.
• Immersive College of Winterhold has 14 actual conflicts with the other mods.
• Morskom Estate has 16 actual conflicts with the other mods.
• Vjarkell Castle has 2 actual conflicts with the other mods.
• Windstad Mine has 10 actual conflicts with the other mods.

Only 48 (24 vs. 24) of the 406 conflicts reported by Mod Organizer are actual conflicts that could impact the player in the game.

 

Duplicates should be removed from conflict detection and the Conflicts tab.

Edited October 26, 2014 by fireundubh

[Tannin]

I doubt this would be practical.

Right now MOs bsa parsing is fast because it reads only the "file records" of bsas to figure out the names and sizes of files, it doesn't read the data.

Calculating hashes for all the files could add minutes to the startup time of MO.

[Aiyen]

Perhaps add it as a function you can activate after startup? I see the point that it would not make any sense to have that sort of indepth analysis running at all times, however it would be nice to have such a tool for general testing. 

[fireundubh]

I doubt this would be practical.

Right now MOs bsa parsing is fast because it reads only the "file records" of bsas to figure out the names and sizes of files, it doesn't read the data.

Calculating hashes for all the files could add minutes to the startup time of MO.

Currently, based on the above test, 80-90% of the conflicts reported are duplicates, which severely limits the usefulness of MO's conflict detection system. Since BSAs pose a problem for hash generation, I'd just calculate hashes for files belonging to non-BSA mods when the user clicks a "build index" button. Save the index as a blob and let the user know when the index needs to be rebuilt. I'd also store the file modified date, file size, and image resolution in this index. MO could use that data to tell you whether a conflicting asset is newer or older, larger or smaller, or a higher or lower resolution, in addition to improving the accuracy of conflict detection overall.

 

[Tannin]

Currently, based on the above test, 80-90% of the conflicts reported are duplicates, which severely limits the usefulness of MO's conflict detection system. 

 

Debatable.

a) In all your examples the mods did conflict, it's just that they conflict in fewer cases. But since you usually resolve conflicts on a per-mod basis (especially when using bsas) and not per-file, that doesn't really matter.

The information "mod a conflicts with mod b" is what's useful as it tells you there is a conflict to resolve. Which individual files differ is mostly relevant to make up your mind.

b) if two mods contain the same file that probably means one of two things: Either the mod author is a moron and packaged files that are already part of a dependency and thus inflated the file size without need or he's copying files from a "modders resource", which is also available to others. In this case his copy represents a certain state of that resource which, in your case, happens to be the same state as the one in some other mods. However, the next time you update either mod, that shared resource may be different versions of that file.

In short: The fact two files in two mods are the same at one point doesn't mean they will be the same after the next update. Unless you plan to go through all conflicts after every update you're going to make a decision "I want to use files from this mod over files from that mod independent of the version" and be done with it.

In this case the information "the two files are the same right now" is not very relevant.

 

 

 

Since BSAs pose a problem for hash generation, I'd just calculate hashes for files belonging to non-BSA mods when the user clicks a "build index" button. Save the index as a blob and let the user know when the index needs to be rebuilt. I'd also store the file modified date, file size, and image resolution in this index. MO could use that data to tell you whether a conflicting asset is newer or older, larger or smaller, or a higher or lower resolution, in addition to improving the accuracy of conflict detection overall.

MO is almost stateless in that it has  no caches or databases apart from profiles and ini files to ensure those don't degenerate/break and to ensure you can work with foreign tools outside MO without breaking MOs stored data.

I've always avoided storing any sort of complex data that users can't read/interpret manually, MO is complex enough.

Your proposal would break this: Use a foreign tool to change/update a mod and MO will not know it needs to update its index.

I don't intend to add a new dimension of user problems "MO doesn't display conflict information correctly because it's index is corrupted" or "MO didn't tell me about this conflict because I forgot to manually re-build the index"

 

Apart from that: I won't implement such a feature if it works only for people who extract BSAs because I don't want people to extract their bsas!

[fireundubh]

A duplicate is a conflict in name only. Duplicates have no impact on the game whatsoever. When 80-90% of the so-called conflicts have no impact on the game, MO offers nothing more than a placebo. "Mod A conflicts with Mod B [in some way]" is not useful information. How can you even say that? You're a software engineer. What if I told you that "Mod Organizer doesn't work on Windows 9"? You'd ask, "What do you mean by 'doesn't work'?" The information that MO provides re: conflicts is a good first step but that's all that it is. 

In short: The fact two files in two mods are the same at one point doesn't mean they will be the same after the next update. Unless you plan to go through all conflicts after every update you're going to make a decision "I want to use files from this mod over files from that mod independent of the version" and be done with it.

No, that files can change is irrelevant to the user at that time they're sorting out conflicts. Mods do not miraculously change without user intervention. When a user adds or updates a mod, all the user would need to do is rebuild the index, assuming the user wants to identify and resolve conflicts. Anyone who has played Skyrim with mods knows that they have to run all sorts of programs after every single change, from LOOT to Wrye Bash to FNIS to any growing number of SkyProc patchers and so on. There's a cost to modding—time spent resolving conflicts—so real conflict detection that actually works is priceless. 

In this case the information "the two files are the same right now" is not very relevant.

That's super important information! To anyone looking at their set of mods for conflicts, the state of their set of mods *right now* is what matters. 

MO is almost stateless in that it has  no caches or databases apart from profiles and ini files to ensure those don't degenerate/break and to ensure you can work with foreign tools outside MO without breaking MOs stored data. I've always avoided storing any sort of complex data that users can't read/interpret manually, MO is complex enough. Your proposal would break this: Use a foreign tool to change/update a mod and MO will not know it needs to update its index.

You're the MO expert. You know best about implementation. I'm just speculating on how a solution might be found. 

I don't intend to add a new dimension of user problems "MO doesn't display conflict information correctly because it's index is corrupted" or "MO didn't tell me about this conflict because I forgot to manually re-build the index"

1. Don't write an index that gets corrupted! ;)2. Notify users when they need to rebuild the index. 

Apart from that: I won't implement such a feature if it works only for people who extract BSAs because I don't want people to extract their bsas!

Well, BSAs don't give you a whole lot of information about the files within them, so unless there's a way to uniquely identify those files and there might be, there isn't much of a choice.Maybe the solution is improving MO's implementation of conflict detection, or maybe the solution is a separate app that runs through MO.In any case, it'd be nice if you didn't think the worst of all of my suggestions.

[Tannin]

I gave you my opinion on your proposal, I explained why I'm not doing it, I could have just said "no" and be done with it.

 

 

 

In any case, it'd be nice if you didn't think the worst of all of my suggestions. 

Look, your suggestion involves me pushing out other projects, investing a lot of my free time I could invest on other things.

If you're going to take it personally when I question your suggestions critically you may want to stop making them.

Honestly, I appreciate proposals but please don't expect you can order stuff and I'll invest hundreds of hours of my time just because you claim it's useful or because you asked nicely.

If I were to implement everything anyone has ever said they'd find useful I'd already have enough work on my plate for the next 20 years.

It's not enough that you think this is useful. It's not even enough if you convince me that it's useful. You'd have to convince me that it's more useful than all the other proposals that will keep me busy for years.

Be more convincing, be less miffed. ;)

Or be helping. Maybe if I didn't have to develop everything myself I'd have time for less pressing issues.

[z929669]

I see an opportunity for independent MO plugin development here ;)

[fireundubh]

edit: Originally posted in Guides & Tutorials.

 

Mod Organizer's conflict detection features are, for all practical purposes, BROKEN. The conflict flags don't indicate actual conflicts; they merely suggest the possibility of conflicts. The Conflicts tab is just as useful, spitting out non-actionable data at the file level. You can assume that 80-90% of the reported file conflicts are false positives.

 

Tannin has stated that he will not fix this problem.

 

Therefore, it is up to you to put in the effort to identify conflicts between your mods.

 

In order to identify real conflicts between your mods:

• Download and install the File Checksum Integrity Verifier from Microsoft to a system folder or add the location of FCIV to your system PATH variable.
• Determine which mods you want to compare for conflicts.
• Extract the respective BSAs, if any, to their respective mod folders but first account for loose files. You want loose files to overwrite anything extracted from the BSAs.
• Generate an MD5 hash index for each mod. On a command line, run: fciv -r -md5 %directory% > %modname%.txt (where %directory% is the path to a mod's folder and %modname% is the name of that mod; if there are spaces in the path or name, enclose the path or name with quotes)
• Repeat that command for each mod you want to compare. (Note: You can generate an index for the entire mods folder but that process is more involved.)
• Open each %modname%.txt in a text editor.

You'll see content that looks like this:

450eca558d9c60bb31f37e18a9836197 locations - heljarchen farm\HeljarchenFarm.esp
ecf16f79575439b08f3b8a7cdabd558b locations - heljarchen farm\meshes\aars\bookset\bookset01.nif
28efe0958121f0ecc0022057084e48fe locations - heljarchen farm\meshes\aars\bookset\booktower.nif
709e9ff1f9e0d58b026bff222b90837c locations - heljarchen farm\meshes\aars\bookset\booktowermini.nif
8599dc9fc4e7472f3456682166dfacfe locations - heljarchen farm\meshes\aars\foodcontainer\basket03apple.nif

• Add tabs between the MD5 hash and the file path. Add tabs after the parent folder in the path (e.g., "locations - heljarchen farm").
• Copy the file listing to an Excel spreadsheet (or open source equivalent.)
• Add headings to the spreadsheet: MD5 Hash, Mod, File Path.
• Apply conditional formatting to the File Path column: Conditional Formatting > Highlight Cells Rules > Duplicate Values (Light Red Fill)
• Apply a color filter to the File Path column: Filter > File Path > Filter By Color (Light Red Fill)
• Copy the filtered results to a new spreadsheet named "Conflicts and Duplicates."
• On the new spreadsheet, remove duplicates: Remove Duplicates > Check: MD5 Hash and File Path
• On the new spreadsheet, apply a color filter to the File Path column again: Filter > File Path > Filter By Color (Light Red Fill)
• Remove the rows where the File Path equals "meta.ini." (Mod Organizer users only)

You now have a list of actual conflicts!

 

Examples:

daf3c0244aa0075791ae06856e9ff00b locations - morskom estate\meshes\food\cratemead01.nif
e890dfa2e376509fdd3c12d58a708716 locations - heljarchen farm\meshes\food\cratemead01.nif

b516833efee0393a75a64e1d73034791 locations - windstad mine\meshes\treasures\trjewelbox01.nif
c134d0af83c7ac607322a51a8e64f860 locations - morskom estate\meshes\treasures\trjewelbox01.nif

58b5af8918a0a95ecbeb36d98b339e10 locations - morskom estate\textures\aars\openbook\largebookpaper04.dds
fb1e370aed8ae94e47642afef8b89a24 locations - immersive college of winterhold\textures\aars\openbook\largebookpaper04.dds

Note that some of these conflicts might just be the difference between an optimized texture and an unoptimized texture.

 

You'll need to compare the files individually, or write a script that does a file size, file modified date, and/or image resolution comparison on every file in the list.

Edited October 28, 2014 by fireundubh

[z929669]

Good info, thanks

 

... but bashing MO functionality --given the wealth of functionality already implemented-- seems a bit harsh.

 

I understand the value in knowing if a conflict is a redundancy versus a difference (both are conflicts), but just knowing there is a conflict is still useful. I like your suggestions, fire, but I also understand Tannin's unwillingness to implement.

 

I would MUCH rather be able to copy all of my conflicts as plain text from the conflicts tab of the Information window. I can CTRL+c each line, but that is a PITA.

 

Wrye Bash is a great model for conflict resolution and working with plain-text lists like this (but does WB differentiate redundant conflicts from differential conflicts? ... I thought not)

[fireundubh]

... but bashing MO functionality --given the wealth of functionality already implemented-- seems a bit harsh.

I'm strictly speaking of conflict detection which, practically speaking, is broken when 80-90% of the reported conflicts are false positives.

 

I understand the value in knowing if a conflict is a redundancy versus a difference (both are conflicts), but just knowing there is a conflict is still useful.

The first step is recognizing you have a problem, but you still need to know what that problem is and how to solve that problem; MO doesn't help you at all in that regard. MO's ability to detect conflicts is a bit like this math problem: "There are two trains running in opposite directions. One is moving at 200 mph, the other is moving at 300 mph, and there are several people standing somewhere on both tracks. When will the two trains be parallel?"

 

I would MUCH rather be able to copy all of my conflicts as plain text from the conflicts tab of the Information window. I can CTRL+c each line, but that is a PITA.

I was going to suggest an "export report" button for the Conflicts tab, so that you could save the file listing (with junk data) to a CSV file, but I'd much prefer if MO's conflict detection features made an effort to remove junk data from the results. Sadly, that's not going to happen, apparently. "File Size," "Last Modified Date," and "Resolution" columns would be great for that report, too.

 

You'd still need to generate MD5 hashes for each file in the CSV. MO couldn't do that on export because that might involving extracting BSAs, but having a premade "Conflicts and Duplicates" spreadsheet would eliminate at least nine of the steps above and make writing an automated conflict detection tool much easier.

Edited October 28, 2014 by fireundubh

[z929669]

Your wish list is desirable, but idealistic. I do see there being a big problem though with processing time, MO side. Again, I have no good knowledge of how deeply Wrye Bash delves into BSAs beyond extracting path information, but my guess is that it does not. That application does to a real CRC check, but again, it still only lists the path in the conflicts tab. Other tabs do hold more information though about a particular file (inside BSAs?, I thought not).

 

I would like to see this kind of functionality too as an option, but I also can see why it would be burdensome to implement when the main goal of MO (I am supposing) is to make it as simple as possible for people to install mods (for many fast-switching profiles) using guide instructions (nothing more).

 

MO would be a more attractive modder's resource if it had some of the conflict resolution functionality you speak of, definitely. I just have to assume that is not the primary audience and functionality.

 

Therefore, it really would be nice if other developers were willing to create some much-wanted plugins for MO to handle these things.

[phazer11]

Fireundubh I'm sure if you asked; someone could help you make a plugin to achieve at least some of the functionality you're wanting. Tannin is maintaining Mod Organizer on his own time, with little to no help so adding one more thing for him to do is a bit much to ask, since the rest of the functionality works just fine for everyone else.

Uh... ninja'd.

[fireundubh]

MO would be a more attractive modder's resource if it had some of the conflict resolution functionality you speak of, definitely. I just have to assume that is not the primary audience and functionality.

This is not about modders... This is about users. Do you not understand the problem? See below.

 

since the rest of the functionality works just fine for everyone else.

We've all been under that impression, certainly, but it turns out that's just not true.

1. The whole point of using MO is to install mods without screwing around in the game installation directory.
2. Some mods that would otherwise need to be installed in a certain order can be simply reordered in MO. That's what the Priority column is showing you.
3. You use the conflict flags in the Flags column, and the Conflicts tab, to determine how to reorder mods.
4. But since 80-90% of the reported conflicts are duplicates, you cannot rely on anything that MO tells you about conflicts.
5. Therefore, MO does not provide you with actionable information that actually helps you determine how to reorder mods.
6. Therefore, the whole point of using MO—for everyone!—suffers from this oversight.

Screenshots:

 

You can't rely on any of the conflict flags in the Flags column. The flags are updated based on the same data available in the Conflicts tab.

 

 

The Conflicts tab shows you all "conflicts," which includes duplicate files that don't matter to anyone.

 

This window is telling you one thing and one thing only: "there might be conflicts between this mod and others. Maybe."

 

Edited October 28, 2014 by fireundubh

[z929669]

I disagree with that last post (but I admit not having read through this thread from the beginning) ... I don't even look at those flags! (previous Wrye Bash user here). All I do is install STEP according to the instructions (even though I helped create many of them). I look at the order of installation, and all of the mod notes and install the mods.

 

Somebody else has done the work of determining differential conflicts from redundancies, so I don't have to ...

 

The people that could really use the true conflict res you speak of are the people creating the guide

BUT, I don't spend time checking the work of others' testing the before/after effects of the mod. I have to trust that they know what overwrites what and the actual in-game effect is accurate and judge based on the screens. I might only do this for one or two mods in a release, but some people do several, which is a big job.

 

MO tells us what overwrites what, and we specifically test this against that, so there is just no real issue with knowing if the assets of two mods are all actually different in some way, because we go by info (including visual) passed on by those before us, and we need to look at the in-game effects of the thing anyway. It would be nice to know when you don't need to review a particular texture (most conflicts are textures), but who has that kind of time anyway? We look at the high-level effects and a few random specifics and don't focus in on every single change in-game. The added conflict resolution functionality would be useful for deeper analysis at times though or for plugins, scripts and other things you can't really 'see' in the game.

 

... and I just want to extract --via text copy-- the text information of any list in MO. The mod list, the plugin list ... the conflicts lists. This is useful for the specialized task of sharing information when cooperatively troubleshooting or testing a mod.

[fireundubh]

I disagree ... I don't even look at those flags! (previous Wrye Bash user here). All I do is install STEP according to the instructions (even though I helped create many of them). I look at the order of installation, and all of the mod notes and install the mods. Somebody else has done the work of determining differential conflicts from redundancies, so I don't have to ...

You're speaking as a pack user, which makes sense given the venue, but most people don't use a pack of any kind.

 

I have to trust that they know what overwrites what and the actual in-game effect is accurate and judge based on the screens.

I doubt any of the pack creators have gone to the lengths that I described above. I'd bet that pack creators "resolve" conflicts by a process artists generally refer to as "eyeballing."

 

MO tells us what overwrites what, and we specifically test this against that, so there is just no real issue with knowing if the assets of two mods are all actually different in some way, because we go by info (including visual) passed on by those before us, and we need to look at the in-game effects of the thing anyway. It would be nice to know when you don't need to review a particular texture, but who has that kind of time anyway? We look at the high-level effects and a few random specifics and don't focus in on every single change in-game. The added conflict resolution functionality would be useful for deeper analysis at times though or for plugins, scripts and other things you can't really 'see' in the game.

If you're fine with "good enough," great! I'm not. I don't use any packs. As you can see from the scrollbars in my screenshot of MO, I have around 200 mods installed, including some rather large overhauls. MO reports that 70 of those 200 mods conflict, which probably equals several thousand files. To get everything working together, I need to be really certain about whether the reported conflicts are serious.

Edited October 28, 2014 by fireundubh