Improving Conflict Detection by Removing Duplicates

[fireundubh]

I have installed a number of location mods that appear to have conflicting assets.

• Heljarchen Farm
• Immersive College of Winterhold
• Morskom Estate
• Vjarkell Castle
• Windstad Mine

I extracted each BSA and generated MD5 hashes for every file in each directory.

 

I imported the results into a spreadsheet and confirmed that the totals below match Mod Organizer's results.

• Heljarchen Farm has 74 conflicts with the other mods.
• Immersive College of Winterhold has 49 conflicts with the other mods.
• Morskom Estate has 151 conflicts with the other mods.
• Vjarkell Castle has 2 conflicts with the other mods.
• Windstad Mine has 130 conflicts with the other mods.

Mod Organizer finds a total of 406 conflicts between these mods but most of these conflicts aren't meaningful.

 

When I removed the duplicates and filtered for actual conflicts (same file names, same file paths, different MD5 hashes), there was a dramatic difference in the number of conflicts.

• Heljarchen Farm has 6 actual conflicts with the other mods.
• Immersive College of Winterhold has 14 actual conflicts with the other mods.
• Morskom Estate has 16 actual conflicts with the other mods.
• Vjarkell Castle has 2 actual conflicts with the other mods.
• Windstad Mine has 10 actual conflicts with the other mods.

Only 48 (24 vs. 24) of the 406 conflicts reported by Mod Organizer are actual conflicts that could impact the player in the game.

 

Duplicates should be removed from conflict detection and the Conflicts tab.

Edited October 26, 2014 by fireundubh

[z929669]

Packs?

 

No, the STEP Guide. I don't even help with Packs. We are far too consumed with maintaining the site and testing mods for STEP (when I can). Most conflicts are texture conflicts, so nothing serious can happen with that other than a borked game asset. Then you just use Mfg Console to determine the source.

 

You are a mod author and are looking to run a bunch of mods together in a single go it sounds like. I do it more slowly and repetitively I guess, one mod at a time. Just knowing the mod you are installing tells you most of what you need to look out for and what you don't want to be overridden. I am not saying the functionality you want is not reasonable ... I would really like to see it, actually. I am just saying that it is by no means essential, since we (I?) have been getting by without it since 2003 or so.

[fireundubh]

Packs? No, the STEP Guide. I don't even help with Packs. We are far too consumed with maintaining the site and testing mods for STEP (when I can). Most conflicts are texture conflicts, so nothing serious can happen with that other than a borked game asset.

I see the STEP Guide as a kind of pack—a guide to building a pack without the tedious annoyance of dealing with modders who want to be permission givers and see their silly names everywhere.

 

I am just saying that it is by no means essential, since we (I?) have been getting by without it since 2003 or so.

You shortcut the mod installation process by using and relying on a guide. That's fine, but the UX design of MO clearly meant for users to use conflict flags and the Conflicts tab to order mods. Saying that proper conflict detection isn't essential to MO because you don't use that part of MO is like saying that the claw isn't essential to a hammer because you use only the face. Edited October 28, 2014 by fireundubh

[GrantSP]

@fireundubh

I get where you're coming from, but I really think you're making a mountain out of a molehill.

 

Whenever a mod user installs a mod the onus to install it correctly is on him. The user should know what it is about that particular mod that may/may not result in a 'conflict'. To help him/her do that the mod authors usually provide some sort of information about what the mod contains and any possible problems from mods that also have those files.

 

If I install ModA because it has textures that I want to overwrite vanilla files, that is my choice and I have the responsibility to know which files. If I also see that ModA also shows some 'conflicting' meshes then likewise, it is I that needs to know which ones, not a tool designed to assist me in installing them.

 

Let me ask you, before MO did you have the same or similar set of mods installed? Presumably these 'conflicts' were also there in your previous installs. Did you not notice them? Or perhaps those 'conflicts' were not really conflicts at all. If you had 200 mods installed I assume you did as much testing as needed as you installed them. What's the difference? MO shows 'conflicts' that aren't because the game runs fine and looks ok. When you install mods with any other tool those possible 'conflicts' are still there but you aren't seeing any flags so you don't notice.

 

Sure, if there was a plugin that provided the sort of detail you describe, I would no doubt enjoy seeing it. But as a tool to aid in installation MO does just fine.

[fireundubh]

Sure, if there was a plugin that provided the sort of detail you describe, I would no doubt enjoy seeing it. But as a tool to aid in installation MO does just fine.

I didn't propose a new feature. I proposed improving an existing feature that really doesn't work. MO has been lying to you. All those conflicts that MO reports to you? 80-90% of them have no impact on the game because they're duplicates. I proposed improving the mechanism that MO uses to identify conflicts by removing duplicates from the results.

 

Because this is the reality of MO's ability to report conflicts that the user can do anything about:

 

 

I'm done explaining this, and Tannin said he won't fix it, so stop trying to convince me that the way you use MO is the only way that matters.

[Tannin]

The way you go ballistic because people disagree with you makes it very hard to take you seriously.

I'm done with the topic as well, just one general question I'd really like answered:

Why, if this is so damn important, are you not working on a solution? Why didn't you even ask if/how you could integrate a solution into MO?

 

The answer - I have to conclude - is that this is important enough for you to rant on a forum but not important enough for you to actually invest time on a solution.

In your opinion it's worth my time but not yours.

That is a bit insulting you know?

[fireundubh]

The way you go ballistic because people disagree with you makes it very hard to take you seriously.

Excuse me, but the people who voiced an opinion about my suggestions, excluding you, actually agreed that improving conflict detection is important—albeit just not as important as I think it is because they use MO differently.

 

"It would be nice to have such a tool for general testing."

—Aiyen

 

"I understand the value in knowing if a conflict is a redundancy versus a difference (both are conflicts), but just knowing there is a conflict is still useful. I like your suggestions, fire. [...] I would like to see this kind of functionality too as an option. [...] MO would be a more attractive modder's resource if it had some of the conflict resolution functionality you speak of, definitely. [...] The people that could really use the true conflict res you speak of are the people creating the guide. [...] I am not saying the functionality you want is not reasonable; I would really like to see it, actually."

—z929669

 

"Sure, if there was a plugin that provided the sort of detail you describe, I would no doubt enjoy seeing it."

—GrantSP

 

We disagree about tangential issues related to conflict detection but we all agree that conflict detection should be improved in some way—whether through a plugin, an app, or MO itself. You've ruled out MO. I have no interest in explaining, or arguing about, MO's ability to detect conflicts now that you've closed that door. I wanted to refocus on solving the problem with a plugin or app, but z929669 (I think) moved my tutorial post into this thread.

 

Why, if this is so damn important, are you not working on a solution?

I guess you didn't see that tutorial I wrote up about how to identify actual conflicts using MD5 hashes and Excel? That's a start.

 

I'm also working on a batch script that generates the CSV automatically, along with resolution data retrieved from ImageMagick.

 

Why didn't you even ask if/how you could integrate a solution into MO?

Because you already gave me information on plugin development in June?

Edited October 28, 2014 by fireundubh

[Tannin]

Well then, have fun...

[DoubleYou]

For a good video on conflict resolution, I recommend

 

I find it rather pointless to be observing conflicts between mods that are using the modders' resources for content mods. The only time you should need to be worrying about mesh and texture conflicts should be by your mesh and texture replacer mods and when you find something that looks wrong. Remember the point of mods is to enhance a game. A game. This isn't life or death. Now Tannin's time... that is life or death!

[fireundubh]

I find it rather pointless to be observing conflicts between mods that are using the modders' resources for content mods. The only time you should need to be worrying about mesh and texture conflicts should be by your mesh and texture replacer mods and when you find something that looks wrong.

Meshes and textures aren't the only assets that concern me. They just happen to be the only sources of conflicts in the example I provided.

[DoubleYou]

Meshes and textures aren't the only assets that concern me. They just happen to be the only sources of conflicts in the example I provided.

Well then, if you're referring to scripts then, there shouldn't be much trouble, as those issues become obvious.

[fireundubh]

Well then, if you're referring to scripts then, there shouldn't be much trouble, as those issues become obvious.

How about preventing problems before they become problems? That sounds like a plan to me.

 

The initial console version of my program in C# is almost done. Takes two seconds to generate 2,862 MD5 hashes across five mods. Just a few more bugs to work out.

Edited October 28, 2014 by fireundubh

[Aiyen]

Problems are only problems because everybody is editing on the same base, and in most cases with the same names for files, scripts etc. 

 

I just thought you wanted a feature that would only show the "meaningful" conflicts.. so only the mod you look at and the mod that actually does any overwriting, instead of everything. 

 

Would be a nice touch... but I guess you wanted something a bit more complex, which I do not really see a point in yet, since most of the problematic conflicts are not something MO should be tackling but rather tes5edit. 

[fireundubh]

Problems are only problems because everybody is editing on the same base, and in most cases with the same names for files, scripts etc.

Comparing file names isn't a great way to determine if files are different. MD5 hashes plus other metadata will serve to distinguish files more accurately. 

 

I just thought you wanted a feature that would only show the "meaningful" conflicts.. so only the mod you look at and the mod that actually does any overwriting, instead of everything.

Yup.

 

but I guess you wanted something a bit more complex

Not sure how you got there.

 

Going to sleep.

[Aiyen]

Well I wont pretend to know all about MD5 hashes etc.... so for me that is more complex! 

 

Guess the going to sleep idea is not half bad! :) 

[fireundubh]

I uploaded a very rough and very basic version of my program to GitHub.

 

Please use that thread for further discussion since this is not an MO issue.