fireundubh

Why are your #1s always blank? "Technically it would be easy to discover all esps (plus associated bsas) in the data directory" "Extending the recognition of .esp+.bsa pairs found in the real /data beyond Update.esm/.bsa and the official DLCs, to any third party mods" Not possible without user intervention. ESPs and BSAs aren't inherently associated in any way. Users, or a group of maintainers, would need to manually associate non-core plugins with BSAs. Some of the work could be automated by initially assuming that {filename}.esp is associated with {filename}.bsa, but again, that's not always the case. Nevermind that some mods have multiple BSAs. The user would need to be presented with a list of suggested ESP-BSA associations and choose whether to accept or change them.

Again, this is not necessary and Tannin has already said no to the idea of an inclusion/exclusion list. It is very possible to read the first 9 bytes of every ESP very quickly to determine whether the ninth byte has the ESM flag. In other words, MO can auto-detect ESPs flagged as ESMs and treat those plugins accordingly. To demonstrate how easy this can be accomplished, in Ruby, for example: #!/usr/bin/env rubyrequire 'hex_string'ARGV.each do|plugin| esm = File.read('#{plugin}')[8].to_hex_string if esm == '01' puts '#{plugin} is flagged as an ESM.' endendThe question then becomes, well, what do you do with them once they've been detected? Tannin doesn't want MO to muck around in the Data directory, so that leaves out moving files. Copying files ruffles the feathers of disk space conservationists. And symbolic links cause LOOT to fail. If MO can't manipulate those plugins in any way, what's left? To make matters worse, there isn't a way to determine plugin-BSA relationships. How can MO know that AAA.bsa belongs to AAA.esp? File name? Well, there's no Skyrim.bsa for Skyrim.esm. And, for a time, Beards Unleashed and Beards had the same plugin name, so what would happen if the user had installed Beards Unleashed over Beards, overwriting Beards.esp as a result? If the latter mod had a BSA, it'd then be associated with the wrong plugin. File name comparisons is an easy yet unreliable way to validate relationships. If MO had the capabilities of TES5Edit, MO could grab a list of the assets, along with their paths, used in each plugin and then try matching that list against the file list in each BSA. Despite the amount of work Tannin would need to put in to read plugins, even that method isn't a surefire way to accomplish the task. BSAs always contain the file names of packed assets and usually the paths, too, but not always. So, we're back to comparing file names. If BSAs included checksum data, MO could make this method work, but BSAs don't. And I'm not even touching on how third-party content in the Data directory doesn't necessarily come neatly packaged in ESP/BSA pairs. Remember the days when people extracted loose files into the Data directory? Well, they still do. Therefore, the best available option, in my opinion, is for MO to do nothing with those plugins and BSAs. Absolutely nothing. Force MO to ignore all third-party content in the Data directory and then tell the user what's happening so that they're not completely clueless about why they can't find their mods. In the end, I think the recommendation to new MO users transitioning from NMM/Wrye Bash should be: reinstall Skyrim and leave the Data directory alone!

This error also occurs when you use symbolic links in MO that point to plugins in the Data directory.

When I compare two identical BSAs, one with Retain File Names and one with Embed File Names, in a hex editor, I don't see a material difference. There's nothing different except the flag bytes.I also don't see a material difference between a plain BSA and one with Retain File Names, Retain Directory Names, or Retain File Name Offsets.It's possible that these flags actually do nothing! But with flags like Retain File Name Offsets and Retain Strings During Startup, it's more likely that these flags instruct the game engine on how to load game data from the associated BSAs. Compress Archive and Xbox 360 Archive are probably the only flags that materially change the file.If we could get tangible results from these flags—results that we can test—we could use BSAopt to create pure BSAs and manually flag them with a hex editor to see what happens. If we could produce good results, someone could make a program that "fixes" BSAopt-created BSAs or even a program that goes through your mods and automatically flags them. Yes? Try and report back.

I updated the list with which BSAs have which resource checkboxes checked. I also included a table with every combination of resource. My powers of awesomeness are growing.

BSAopt can produce only uncompressed (0x03) and compressed BSAs (0x07). No flags. They are otherwise identical to Archive-created uncompressed and compressed BSAs.

edit: Nevermind.

See my thread: BSA Archive Flags.There's probably a good reason why Bethesda compressed only the animations, meshes, and textures BSAs.

I used my powers of awesomeness to determine the archive flags that were used to pack the official BSAs. This information could be important! Skyrim - Animations.bsa Flags: Compressed Skyrim - Interface.bsa Uncompressed, Flags: None Skyrim - Meshes.bsa Flags: Compressed, Retain Strings During Startup Resource Types: Meshes Skyrim - Misc.bsa Uncompressed, Flags: Retain File Names Skyrim - Shaders.bsa Uncompressed, Flags: None Skyrim - Sounds.bsa Uncompressed, Flags: Retain File Names Resource Types: Sounds, Voices Skyrim - Textures.bsa Flags: Compressed, Embed File Names Resource Types: Textures Skyrim - Voices.bsa Uncompressed, Flags: None Resource Types: Sounds, Voices Skyrim - VoicesExtra.bsa Uncompressed, Flags: None Resource Types: Sounds, Voices Dawnguard.bsa Uncompressed, Flags: Retain File Names, Retain Strings During Startup Resource Types: Meshes, Sounds, Textures, Voices HearthFires.bsa Uncompressed, Flags: Retain File Names, Retain Strings During Startup Resource Types: Meshes, Sounds, Textures, Voices Dragonborn.bsa Uncompressed, Flags: Retain File Names, Retain Strings During Startup Resource Types: Meshes, Sounds, Textures, Voices HighResTexturePack01.bsa Flags: Compressed, Embed File Names Resource Types: Textures HighResTexturePack02.bsa Flags: Compressed, Embed File Names Resource Types: Textures HighResTexturePack03.bsa Flags: Compressed, Embed File Names Resource Types: Textures Note: By default, all BSAs have these flags: Include Directory Names and Include File Names. You can't change those flags, so I didn't add them to the list. ARCHIVE FLAGS RESOURCE FLAGS

I don't know. I moved all of the nonessential signatures to separate plugins, including the wide variety of possible lzma signatures.I also created a magic signature for BSAs.# BSA archives0 string BSAx00 BSA archive>4 byte 0x67 b, version: %d>4 byte 0x68 b, version: %d>8 byte 0x24 b, folder records offset: %dWhen I run Binwalk on Update.bsa, I now get:DECIMAL HEX DESCRIPTION-------------------------------------------------------------------------------------------------------------------0 0x0 BSA archive, version: 104, folder records offset: 367422937 0x7143D9 Zlib header, best compression, uncompressed size >= 655367688740 0x755224 Zlib header, best compression, uncompressed size >= 461827704627 0x759033 Zlib header, best compression, uncompressed size >= 1020487737852 0x7611FC Zlib header, best compression, uncompressed size >= 655367855149 0x77DC2D Zlib header, default compression, uncompressed size >= 812617880806 0x784066 Zlib header, default compression, uncompressed size >= 1068517913871 0x78C18F Zlib header, default compression, uncompressed size >= 869927942406 0x793106 Zlib header, default compression, uncompressed size >= 9830416855268 0x10130E4 Zlib header, default compression, uncompressed size >= 9830416902615 0x101E9D7 Zlib header, default compression, uncompressed size >= 1757916908715 0x10201AB Zlib header, default compression, uncompressed size >= 9830416955011 0x102B683 Zlib header, default compression, uncompressed size >= 6130616978305 0x1031181 Zlib header, default compression, uncompressed size >= 6553617038589 0x103FCFD Zlib header, default compression, uncompressed size >= 6553617109606 0x1051266 Zlib header, default compression, uncompressed size >= 9830417156267 0x105C8AB Zlib header, default compression, uncompressed size >= 7233817179168 0x1062220 Zlib header, default compression, uncompressed size >= 9830417213805 0x106A96D Zlib header, default compression, uncompressed size >= 9830417261891 0x1076543 Zlib header, default compression, uncompressed size >= 13107217306910 0x108151E Zlib header, default compression, uncompressed size >= 9830417353626 0x108CB9A Zlib header, default compression, uncompressed size >= 65536On Skyrim - Misc.bsa:DECIMAL HEX DESCRIPTION-------------------------------------------------------------------------------------------------------------------0 0x0 BSA archive, version: 104, folder records offset: 36I think that makes clear that the latter file is not compressed, so I can effectively use Binwalk to determine whether BSAs are compressed without extracting them.Probably not that useful, but still (nerdy) cool!

Magic.I linked the same page. That page indicates that there is uncertainty about various elements of the BSA format.And, FYI, I was just trying to answer to EssArrBee.Also, "Skyrim - Misc.bsa" may not be compressed at all. Binwalk finds zlib headers for other BSAs.edit: Binwalk reports many lzma false positives because lzma has no magic bytes for which to create a reliable signature.

Already did. Signature Analysis Results If we could adapt this BSA header information to a magic signature, Binwalk might produce different results, but I'm not sure how useful that would be in the long run. The object of doing all this was simply to determine the type of compression used by Skyrim.

Lzma is the same compression used by 7-Zip. MO extracts .7z archives all the time. Binwalk is a complex binary analysis program. Using its signature analysis feature on the core BSAs, the results suggest zlib headers for some while others are strictly lzma. Some results are false positives. For example, Binwalk finds Ubicom firmware headers in "Skyrim - Misc.bsa." However, of the 113 results, only three are Ubicom firmware headers whereas the remaining 110 are lzma compressed data. No zlib headers are suggested. Since the signature analysis scans the whole file by default, these results could mean that .bsa is lzma compressed data or the .bsa is uncompressed but contains lzma compressed data. In any case, I just found this program yesterday and spent the whole day trying to install Binwalk on CentOS, then installing Debian, and then sorting out dependency issues. So, I'm no expert. Here's what Update.bsa looks like through the lenses of signature and entropy analysis:

In xEdit, EDID queries are super slow relative to FormIDs.

edit: I ran Binwalk on the DLC BSAs. Looks like the BSAs use zlib compression. What version? Beats me. edit 2: Okay, some BSAs use zlib compression; others use just LZMA.

No problem. By the way, I could modify the script so that the various FormIDs are maintained in the exported data. I opted to export only EDIDs because I figured you'd find those more useful for manual comparison. LMK.

Sure, but the entire point of using MO is so you don't do that. Since MO now intercepts Steam Workshop downloads and puts them into Overwrite, there should be no reason to put anything in the Data folder. If you want to protect users from themselves, I'd more readily suggest that MO IGNORE all plugins in the Data folder except the core Skyrim and DLC plugins.

You don't need to do that because the issue at hand only reveals itself when ESPs are flagged as ESMs.Just read the first nine bytes of an ESP and check the last byte. If the last byte is 01, that's a false-flagged ESP. Samsung 840 EVO SSD has a 540 MB/s sequential read speed.DDR3 SDRAM has a 6400 MB/s maximum transfer rate.zlib 1.2.8-6 has a 300 MB/s decompression speed.As this data suggests, the file type matters though.What we need is a decompression test for BSAs.

Purpose: Export AMMO, ARMO, COBJ, MISC, and WEAP RecordsGame: The Elder Scrolls V: SkyrimAuthor: fireundubhVersion: 0.1HOW TO USE:1. In Excel, create four worksheets named: Main, Keywords, Flags, and Components.2. Select any number of records of A SINGLE TYPE (e.g., AMMO, ARMO, COBJ) and apply the script.3. Import each CSV in reverse order into the respective Excel worksheets. NOTE: Reverse order is important because the Main worksheet will contain hyperlinks to the other worksheets and Excel is stupid.4. The Keywords, Flags, and Components links may not be formatted correctly but they will still work. To update the formatting: 4a. Use the arrow keys to highlight the topmost unformatted links cell. 4b. Press F2. 4c. Press Enter. 4d. Repeat until you're finished.5. Tell fireundubh how much you love and adore him for making you this script. Kneeling is optional but encouraged.You can download the script here: https://pastebin.com/raw.php?i=i8LtJ5PU And here's a blank XLSX file with four pre-named worksheets: https://copy.com/oHe3ZnCP4Pb4 edit: I rewrote that script to export .csv files instead of outputting messages.

This is what my script will output now: ----------------------------------- SHEET1 ------------------------------------FormID EDID FULL Rating Value Weight Object Effect Keywords Flags0x00012E4B ArmorIronBoots Iron Boots 10.000000 25 6.000000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link")0x00012E4D ArmorIronHelmet Iron Helmet 15.000000 60 5.000000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link")0x00012EB6 ArmorIronShield Iron Shield 20.000000 60 12.000000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link") =HYPERLINK("#'Sheet3'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-8,4)), Sheet3!$A:$A, 0), 1), "Link")0x00013104 ClothesBeggarHat Ragged Cap 0.000000 1 0.500000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link")0x00013105 ClothesBeggarRobes Ragged Robes 1.000000 1 1.000000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link")0x00013106 ClothesBeggarBoots Ragged Boots 0.000000 1 1.000000 =HYPERLINK("#'Sheet2'!"&ADDRESS(MATCH(INDIRECT(ADDRESS(ROW(), COLUMN()-7,4)), Sheet2!$A:$A, 0), 1), "Link") ----------------------------------- SHEET2 ------------------------------------FormID Keyword Keyword Keyword Keyword 0x00012E4B ArmorBoots ArmorHeavy ArmorMaterialIron VendorItemArmor0x00012E4D ArmorHeavy ArmorHelmet ArmorMaterialIron VendorItemArmor0x00012EB6 ArmorMaterialIron ArmorShield VendorItemArmor0x00013104 ArmorClothing ClothingHead VendorItemClothing0x00013105 ArmorClothing ClothingBody ClothingPoor VendorItemClothing0x00013106 ArmorClothing ClothingFeet VendorItemClothing ----------------------------------- SHEET3 ------------------------------------FormID Flag 0x00012EB6 Constant HiddenFromLocalMap BorderRegion HasTreeLODIn Excel, you'd copy Sheet3 first, then Sheet2, and then Sheet1, so that the hyperlinks in Sheet1 resolve without manual intervention. I have all record types done except COBJ. edit: Done. I'm going to implement a way to add Keyword/Flag column headings dynamically, so that if a record has 10 keywords and there are 10 keywords at most in the selected records, then there'd be 10 Keyword column headings in Sheet2. edit: Done. edit: I also made the reference to the FormID cell relative, so that it's no longer hardcoded (i.e., you don't have to copy the header anymore.)

If you're using Excel, I can also separate keywords by newlines in a cell. Let me know. Example: ="ArmorClothing"&CHAR(10)&"ClothingHands"&CHAR(10)&"MagicDisallowEnchanting"&CHAR(10)&"VendorItemClothing"edit: I did one better. In the keywords cell, I've placed a link that you can click to take you to the right row in Sheet2, which would contain the keywords for each Form ID.

Are you sure you don't want keywords on a separate row? You can then copy the keyword rows to a different sheet and then link them. As you can see from the screenshot, there can be a lot of keywords and you'll need a really tiny font to see the whole sheet at 1920x1080.

I have ARMO working: Form ID EDID FULL Weight Value Rating Keywords Flags0x00013105 ClothesBeggarRobes Ragged Robes 1 1 1 ArmorClothing, ClothingBody, ClothingPoor, VendorItemClothing 0x00013106 ClothesBeggarBoots Ragged Boots 1 1 0 ArmorClothing, ClothingBody, ClothingFeet, ClothingPoor, VendorItemClothing 0x000135BA ArmorImperialShield Imperial Shield 12 50 20 ArmorClothing, ArmorMaterialImperialHeavy, ArmorShield, ClothingBody, ClothingFeet, ClothingPoor, VendorItemArmor, VendorItemClothing Constant HiddenFromLocalMap BorderRegion HasTreeLOD0x000136CF ArmorImperialHelmetOfficer Imperial Officer's Helmet 4 30 17 ArmorClothing, ArmorHeavy, ArmorHelmet, ArmorMaterialImperialHeavy, ArmorShield, ClothingBody, ClothingFeet, ClothingPoor, VendorItemArmor, VendorItemClothing 0x000136D4 ArmorImperialGauntlets Imperial Bracers 4 15 10 ArmorClothing, ArmorGauntlets, ArmorHeavy, ArmorHelmet, ArmorMaterialImperialHeavy, ArmorShield, ClothingBody, ClothingFeet, ClothingPoor, PerkFistsSteel, VendorItemArmor, VendorItemClothingWhat other record types did you want? Please be specific. Thanks!

How do you want keywords and flags presented? In a separate row indexed by the Form ID? Or a comma-delimited list in a single field of the same row?

I can write a TES5Edit script to output that data with tabs, so that you can copy and paste that data from the Messages tab into Excel. I've written a ton of scripts, including one really crazy one for Sharlikran. I worked up a simple script to output this data: Form ID EDID FULL Weight Value Rating0001396A ArmorDaedricBoots Daedric Boots 10 625 180001396B ArmorDaedricCuirass Daedric Armor 50 3200 490001396C ArmorDaedricGauntlets Daedric Gauntlets 6 625 180001396D ArmorDaedricHelmet Daedric Helmet 15 1600 230001396E ArmorDaedricShield Daedric Shield 15 1600 36I'll add keywords, flags, and more record types tomorrow.