Looking for documentation behind the concepts of xEdit.

[FullOfChocolate]

I'm struggling with the TES5Edit manual because of my lack of understanding of the key concepts: Records, FormIDs, filenodes, references to base and non-base objects. Oh boy.

I'm getting cranky with these modern Wiki things and... video tutorials and whatnots. They're like jiggsaw puzzles to me, turned upside down. Need to get to the source. I read manuals for fun and cried a little when Mozilla stopped shipping a proper help file with Firefox. And if that was before you were even born... well, you can just get out of my lawn. And please let the dog bite you on your way out.

 

Is there an official, properly constructed "modder's manual" lurking somewhere that the creators of xEdit might have used to build their program? And what resources did you guys use to master this stuff. I can't believe it could've been the Creation Kit Wiki. That, with all due respect, is a mess. Or is the current modding environment such that competence in the field is necessarily limited by the lack of proper documentation.

 

Thank you.

[Mator]

The backbone of TES5Edit is not something the creators of TES5Edit came up with themselves, it's something that's inherent to how Bethesda encodes their files.  TES5Edit was largely reverse engineered from Bethesda files and tools, and has been referred to by the developers as the "Skyrim Plugin Decoding Project".

 

Records, formIDs, filenodes, and references also weren't created by Bethesda.  They trace their roots back to basic data storage concepts and databases.  In the context of databases you'll see the word record (or row) used very often.

 

The Bethesda plugin files themselves are comprised of a bunch of packed records.  What packed means is hard to explain from a technical standpoint, but the result of packing the records (or arrays, depending on the context) is an increase in space and time efficiency.  Like records in most databases, these records have unique IDs.  These IDs are essentially just unsigned integers, and are equivalent to DWords in 32-bit implementations.  That is to say, they're a data type that takes up 32 bits of space, which gives them a maximum value of 2³² or 4,294,967,296 or 16⁸ or $FFFFFFFF (all of these numbers are equivalent).  You'll recognize the final representation of this number as the maximum FormID possible in any Bethesda game.  FormIDs are just hexadecimal representations of the ID associated with a record (or form).

 

Form is roughly synonymous with record.  More technically speaking, you could consider a Form to be a type of record (in the database sense) with certain header information as defined by bethesda (record flags, record signature, formID among those).

 

Your reference to FileNodes seems out of context.  As a TES5Edit user I can imagine no reason why you'd be concerned with that.  If you can clarify what you mean maybe I can help.

 

Regarding references.  A reference is just what it sounds like it is - a reference.  It's like a pointer in programming.  You basically have an address which tells you where another object is so you can make relationships between various records.  In this case, the address is the record's FormID.  The difference between base and non-base records is more esoteric.  To understand this we have to delve into the concept of a load order.

 

So, from what we've said so far, we've established there are records, and that we can have up to about 4.3 billion records in the game (barring memory limitations, of course).  The thing is, we might want to separate our records into logical units (or for users to be able to create new logical units with records in them) in the future.  For Updates, DLC, or Mods.  This is where the notion of a file and a load order prefix comes into play.  If we have more than one file now, we have to make sure that all of the content in those files lines up and that cross-file references will work.  This requires us to be absolutely certain that a FormID isn't taken already.  If each file just arbitrarily assigned FormIDs in the 4.3 billion record space, there'd be no guarantee that File A and File B wouldn't try to make completely different records at the same FormID and thus cause serious issues to occur.  Thus we reserve the first two hexadecimal digits (a space of $FF = 256 = one byte) for an affix that will be assigned to files based on the order that they are loaded into the game.  This makes certain that no two files will try to create records at the same FormID when the game runs, as the Record's FormIDs will be allocated at execution time to prevent conflicts.  This also allows us to set a priority order for changes, so two files that change the same thing can be ordered so one of them wins out over the other.

 

Ok, so we have a load order now.  The next question is why do we have override/injected records (I think that's your question).  While we allocated our FormIDs to complete different spaces to prevent unwanted conflicts, in so doing we made it so patching is impossible.  However, it is extremely valuable to implement patching in a way that doesn't require modifying the original files, as it makes installation and management much easier.  This is where master files, override records, and injected records come into play.  An override record is a record that has been allocated outside of the file's FormID space and that collides with another existing record's FormID.  That is to say, records in a file are all initially allocated to FormIDs in the range $xx000001-$xxFFFFFF ($xx being the number of masters the file has).  These are defined as the FormIDs IN that file.  NOTE that these FormIDs will be mapped to $yy000001-$yyFFFFFF when the file is loaded into the game, with $yy corresponding to the file's load order position.

 

In contrast, FormIDs at $00000001-$00FFFFFF would be considered as referring to FormIDs in the first "master" of the file.  So if I have a file "Test.esp" which has Skyrim.esm as it's master, records that are part of Test.esp will be in the $00000001-$00FFFFFF range and records that are being put into Skyrim.esm are in the $00000001-$00FFFFFF range.  If any record already exists in Skyrim.esm at the specified FormID it will be "overridden" by the version in Test.esp, else what happens is what's called "injecting records".  Similar to how FormIDs in the file get mapped based on the load order position of the file, FormIDs that refer to a master get mapped to the load order of the master.  So $00000001 in Test.esp gets mapped to $00000001 in Skyrim.esm because Skyrim.esm is at load order position 0 (thus not a particularly good example).

 

I'd imagine you're probably pretty confused now, but there's the information dump you seemed to be asking for. 

 

 

Cheerio.

-Mator

Edited June 20, 2015 by Mator

[alt3rn1ty]

A lot of this is also learned from the Bethesda forums

 

Historically a lot of the game specific terminology comes from the official modding tools topics on BGS forums, like the Construction Set for Morrowind and Oblivion, and Creation Kit for Skyrim .. in the sub forums here https://forums.bethsoft.com/forum/13-the-elder-scrolls/

 

There are also some sticky topics which are hugely informative no matter how old they are, because a lot of the subject matter still applies, for example a lot of the material in Oblivion Mods forum, the sticky "Oblivion Mods FAQ" https://forums.bethsoft.com/topic/449239-oblivion-mods-faq/ 

 

.. is still useful for anyone starting out modding Skyrim

 

To catch up on terminology that a lot of us have become familiar with over time ( for me personally I have been frequenting the official forums since 2005, and other support forums for Morrowind long before joining the official support forum ), will probably take a quite a bit of time, but you will pick it up with experience.

Edited June 20, 2015 by alt3rn1ty

[zilav]

Small fix to file FormIDs (numbers that are stored in plugin):

 

".. records that are part of Test.esp will be in the $01000001-$FFFFFFFF range and records that are being put into Skyrim.esm are in the $00000001-$01FFFFFF range"

 

First number is the index of master file in TES4 plugin header starting from 00 (overrides), any numbers above the index of the last master are new records added by a plugin. If first number of FormID is matching the index of some master file but record with that FormID doesn't exist there, it is injected record.

[alt3rn1ty]

Beware some old terminology though which may have been superceeded ( new knowledge / methods / game engine )

 

One example is Archiveinvalidation - Depending on the game there are various methods and recomendations - Oblivion for example has three methods, but the best of those would be BSA Redirection ( as used by Wrye Bash ) for that game ..

 

.. Whereas for Skyrim, its not longer a matter for concern. Same as time stamps on files which at one point was important .. now load order is all you need to get correct. See BSAs and You sticky https://forums.bethsoft.com/topic/1354395-update-bsas-and-you/

 

And tools were very well developed for Oblivion, but for Skyrim they are still playing catch-up ( apart from TES5Edit which is superb )

 

No one site or topic covers everything you need - So feel free to contribute to the community and make one :D

[Mator]

Small fix to file FormIDs (numbers that are stored in plugin):

 

".. records that are part of Test.esp will be in the $01000001-$FFFFFFFF range and records that are being put into Skyrim.esm are in the $00000001-$01FFFFFF range"

 

First number is the index of master file in TES4 plugin header starting from 00 (overrides), any numbers above the index of the last master are new records added by a plugin. If first number of FormID is matching the index of some master file but record with that FormID doesn't exist there, it is injected record.

 

Thanks for that correction.  I'm amazed I got everything else right.  :o

[GrantSP]

STEP is currently in discussion about this very subject and we will endeavour to provide a guide that covers as much as possible for all the xEdit variants in the near future. (participation in this project is not restricted to STEP admin though and all are encouraged to join in if they can)

[Greg]

If you want a deep dive into the TES5 mod file format, you can find it documented at Tes5Mod: Mod File Format. This isn't for the faint of heart, and I mention it only because  I personally get a much better understanding of things by poking around the raw data to learn how to take it apart and put it back together. Unfortunately, some of the records (notably NAVM) haven't been documented anywhere to my knowledge and some of the information may still be a bit confusing. I've edited some of it to hopefully try to clarify what I found when breaking it apart.

 

I still need to finish going through this to finish correlating the raw data with what's shown in the Creation Kit, but I need to find some time to get back to it.

[FullOfChocolate]

Sorry for the late reply. I had problems with the computer.

 

I had this wonderfully naive thought that modding was actively supported by Bethesda. Meaning lots of deep documentation and tools. The mod file format documentation looks useful though. I had trouble understanding what kinds of records and fields there are. And how they're represented in TES5 of course. Seeing how someone could come up with all that by just taking something apart does give me hope :) For a start it might be easiest to try to take a compatibility patch and see how one of those is put together.

 

Thanks to you all for the help.