Better Conflict Discovery Project

[fireundubh]

@fireundubh is currently MIA and has also removed all his projects from all the locations he had them.

As such this page and the links on it are useless.

 

Page is now locked. -- @GrantSP

 

 

"md5gen" accurately discovers conflicts between files.

md5gen was developed with Skyrim and the Skyrim Mod Organizer in mind, but there are probably other applications.

Source available at GitHub | README

Refer to the README for up-to-date information about the program.

I will use this space to outline/discuss modding-specific ideas, concepts, and features.

Why do we need this program?
There are no tools available that accurately report conflicts or duplicates between mods.

• With more information about conflicts, players can better sort their mods or install them in the optimal order.
• With more information about conflicts, modders can easily identify how their mods affect other mods.
• With more information about conflicts, STEP guide authors can write better guides.
• With more information about duplicates, players will benefit from smaller archive sizes, faster downloads, and less conflicts reported by other software.
• With more information about duplicates, modders can easily discover which files are shared across mods or others, and using that information, they can create BSA resource packs containing just those files.

Doesn't Mod Organizer already report conflicts?
Mod Organizer reports too many conflicts. 90% of the conflicts reported in MO are duplicates. Duplicates have no impact on your game, so trying to resolve those conflicts is a waste of time. In addition, because only 10% of the conflicts are meaningful, you're just guessing when you use that data to make sorting decisions.

This program takes the guesswork out of the process and shows you not only which files conflict but also how they conflict.

Later, I'll implement features that will allow you to resolve those conflicts directly from the user interface.

Sample Output - Report Type: Conflicts - Hash Type: CRC32

858A980C|Locations - Morskom Estate|\meshes\treasures\trjewelbox01.nif|730,227
80D78342|Locations - Windstad Mine|\meshes\treasures\trjewelbox01.nif|761,148
1993528C|Locations - Immersive College of Winterhold|\textures\aars\openbook\largebookpaper04.dds|174,904|512|512
1586D9CB|Locations - Morskom Estate|\textures\aars\openbook\largebookpaper04.dds|349,680|512|512

Sample Output - Report Type: Duplicates - Hash Type: CRC32

0109651A|Locations - Immersive College of Winterhold|\textures\aars\openbook\largenote01.dds|174,904|512|512
0109651A|Locations - Morskom Estate|\textures\aars\openbook\largenote01.dds|174,904|512|512
01C88191|Locations - Heljarchen Farm|\meshes\decoration\ruinsflowervase01.nif|209,994
01C88191|Locations - Morskom Estate|\meshes\decoration\ruinsflowervase01.nif|209,994
01C88191|Locations - Windstad Mine|\meshes\decoration\ruinsflowervase01.nif|209,994

• Column 1 is the MD5/CRC32/Murmur3 hash of the file. This fingerprint uniquely identifies the file.
• Column 2 is the name of the mod. In Mod Organizer, the name of the mod is the folder in which that mod resides.
• Column 3 is the path to the file in that mod's folder.
• Column 4 is the size of the file in bytes.
• Column 5 is the width of the file, if the file is a DDS texture.
• Column 6 is the height of the file, if the file is a DDS texture.

The layout will change as I update the program. For example, there's no column for file modified date yet.

Packed vs. Unpacked BSAs
 
I'm not sure if there's a solution to the BSA problem. It might be possible to generate MD5 hashes for each file in uncompressed BSAs. I haven't investigated yet.
 
So, for the time being, BSAs will need to be unpacked for this program to work.
 
Contributing
 
I'm really not that great with C#, so if anyone here is great with C# and wants to contribute, the source is on GitHub under the GNU GPL v3.0 License.

Edited November 23, 2015 by GrantSP

[z929669]

Thanks for providing this tool

[fireundubh]

Updated.
 
You now have more control over which directory you want to analyze and which file you want to output the report to, so you can run the program from anywhere.
 
This update eliminates hardcoded directories; although, you can run the program without the -i and -o arguments, and the program will analyze the current directory and output the report to "output.txt" in that same directory.
 
This update also forces the program to skip files in the current directory, so only subdirectories in the current directory will be recursively analyzed.
 
To compile, you now need to install the Command Line Parser Library. I'd recommend putting the library DLL in the system PATH, although that shouldn't be necessary. Might just be the way I run the program.
 
By the way, on my system, processing 55.4K files (10.4 GB) took 4 minutes and 11 seconds.

Edited October 31, 2014 by fireundubh

[fireundubh]

Updated. I wrote my own command line parser to remove the dependency. No more DLLs. See the README for how to use the program.

 

Also, I opened the door to more report types. The next time I post about an update, you'll probably be able to generate a report of the duplicates between mods.

[fireundubh]

Updated.

 

I haven't yet implemented the duplicates report feature yet, but I added two more parameters: x for excludes and h for hash type.

 

With the x parameter, you can manually exclude a file or any number of files delimited by commas. "meta.ini" is excluded by default, currently.

--exclude file1.ext,file2.ext,file3.ext

With the h parameter, you can choose to use CRC32 instead of MD5. It's a bit slower but looks cleaner. It's also what LOOT uses.

--hash crc32

Parameters also no longer require dashes, double dashes, or slashes, but you can use them and any combination of them.

md5gen -o conflict_report.txt r conflicts /i E:\mods --hash crc32

When you run the program, this is what you'll see now:

E:\mods>md5gen -o conflict_report.txt r conflicts /i E:\mods --hash crc32
-------------------------------------------------------------------------------
Generating conflict report... Please wait.
Saving to: E:\mods\conflict_report.txt
-------------------------------------------------------------------------------
Excluding: \meta.ini (Locations - Heljarchen Farm)
Excluding: \meta.ini (Locations - Immersive College of Winterhold)
Excluding: \meta.ini (Locations - Morskom Estate)
Excluding: \meta.ini (Locations - Vjarkell Castle)
Excluding: \meta.ini (Locations - Windstad Mine)
-------------------------------------------------------------------------------
Operation completed in 00:00:04. Found 40 conflicts.

Edited November 1, 2014 by fireundubh

[GrantSP]

@fireundubh for someone that is "really not that great with C#", you seem to moving along quite nicely. Well done.

[fireundubh]

^Thank you.

 

Updated.

 

Duplicates report generation is now implemented.

E:\mods>md5gen -o duplicates.txt r duplicates /i E:\mods --hash crc32
-------------------------------------------------------------------------------
Generating duplicates report... Please wait.
Saving to: E:\mods\duplicates.txt
-------------------------------------------------------------------------------
-- Operation completed in 00:00:04. Found 360 results.

Sample Output:

0109651A|Locations - Immersive College of Winterhold|\textures\aars\openbook\largenote01.dds|174,904|512|512
0109651A|Locations - Morskom Estate|\textures\aars\openbook\largenote01.dds|174,904|512|512
01C88191|Locations - Heljarchen Farm|\meshes\decoration\ruinsflowervase01.nif|209,994
01C88191|Locations - Morskom Estate|\meshes\decoration\ruinsflowervase01.nif|209,994
01C88191|Locations - Windstad Mine|\meshes\decoration\ruinsflowervase01.nif|209,994

Edited November 2, 2014 by fireundubh

[fireundubh]

Updated.

 

Added support for the Murmur3 hash algorithm. Use "-h murmur3" for Murmur3 hashes instead of MD5 and CRC32.

 

For the technically minded, I intend to implement a byte-by-byte comparison for hashed duplicates to reduce the already tiny chance of hash collisions even further.