Compacting FORMID for ESL

[Guba]

I need a clarification about compacting EPSs with xEdit.
When a plugin is cmpacted and the ObjectIDs renumbered, this doesn't jeopardize the compatibility of the mod with the possible patches or files that use the plugin as master?
I was starting to suspect this when I realize that when xEdit performs a compact, this process involves also other plugins that seem to rely on the plugin being compacted.
An Object ID renumber could put other plugins not finding the objects they are looking for.
I'm completely on the wrong way or is there any sense in what I'm saying?
The doubt came to my mind because I'd seen xEdit to renumbering references also in the files that are related in some way to the plugin I'm compacting.
Sometimes the compating procedure doesn't touch any other plugin, and this way I think is safe, because there is nothing that relies in the information stored in the plugin, but all remains self-contained. Obviously this must be done - if I have understood correctly - with a new game in mind, because the gamesaves may contain references to what "de facto" has changed its "name", resulting impossible to find it back again.

Can someone explain me what is the whole thing of compacting?
I tried to find some info here but it seems that this topic hasn't been covered before.

Thanks for any reply

 

[Mousetick]

4 hours ago, Guba said:

But seems that the limit of FormIDs is 2048, because it seems to be 4096.

Your interpretation of that xEdit script is mostly correct, but you're confusing the count of things with their values. And despite acknowledging DY's distinction between records and FormIDs, you're still apparently confusing between them.

4 hours ago, Guba said:

Since we see this code, the thing to say is that there are TWO conditions to check if a plugin can be flagged as ESL:
 1. the number of records, that must be lower than 2048
2. the number of FormIDs, that must be lower than 4096.

No, that's not right. The script checks these conditions:

1. The number of FormIDs must be lower than 2048.
2. The value of all FormIDs must be lower than 4096.

If condition #1 is met, but not #2, the plugin can still be light by compacting the FormIDs (that is changing their values to be less than 4096).

There is an important piece of code at the beginning, which you noticed and commented on:

if not IsMaster(e) then
      Continue;

This skips over any record that is not a FormID, and only continues processing records that are FormIDs.

To summarize, a light plugin is a plugin with fewer than 2048 FormIDs, each FormID being in the range 2048 to 4095 (FExx x800 to FExx xFFF).

[DoubleYou]

1 hour ago, Guba said:

I need a clarification about compacting EPSs with xEdit.
When a plugin is cmpacted and the ObjectIDs renumbered, this doesn't jeopardize the compatibility of the mod with the possible patches or files that use the plugin as master?
I was starting to suspect this when I realize that when xEdit performs a compact, this process involves also other plugins that seem to rely on the plugin being compacted.
An Object ID renumber could put other plugins not finding the objects they are looking for.
I'm completely on the wrong way or is there any sense in what I'm saying?
The doubt came to my mind because I'd seen xEdit to renumbering references also in the files that are related in some way to the plugin I'm compacting.
Sometimes the compating procedure doesn't touch any other plugin, and this way I think is safe, because there is nothing that relies in the information stored in the plugin, but all remains self-contained. Obviously this must be done - if I have understood correctly - with a new game in mind, because the gamesaves may contain references to what "de facto" has changed its "name", resulting impossible to find it back again.

Can someone explain me what is the whole thing of compacting?
I tried to find some info here but it seems that this topic hasn't been covered before.

Thanks for any reply

 

Don't make esl out of plugins that are required by other mods, or the formids will not match, unless you change them also to match.

Compacting a mod's formids is changing the formids to be in FE space. This requires renumbering of any formids that are New records in the plugin. If all of a mod's records are overrides from previous masters, nothing will be changed. You have a limit of how many formids can be fit in FE space.

See this thread for more information: https://www.afkmods.com/index.php?/topic/5079-plugin-files-and-you-esmeslesp/

[Guba]

Also in plugins that do not require a compacting, there is a limit of 2048 records?
Someone says yes, some says no thanks to the mod ENGINEFIXES.
 

[DoubleYou]

The limit is 2048 formids. The reference cap is not related, and SSE Engine Fixes does not "fix" this, as it is a hard limitation.

[Guba]

I see. So some of the information I gathered from internet is wrong.
Is there any script inside xEdit scripts that counts the number of records massivley?
 

Edited February 14, 2023 by Guba
typo

[DoubleYou]

Rather than keep going on this, I'll just link you this xEdit script that should do what you're after:

https://www.nexusmods.com/skyrimspecialedition/mods/42211

[Guba]

However you maybe wrong.
The proof of my doubts at this point stays in a plugin of the Creation Club AE set which is named
cceejsse003-hollow.esl
that is reported by xEdit containing 2068 records.
This plugin not only is an ESL but it is a ESM too, that is to say a Light Master.
How can be that possible?
Does Bethesda release plugins destined to break the game?
 

[Greg]

New records added to an ESL that do not override records in any masters must be in the range xxxxx800 - xxxxxFFF. Records in an ESL that are just overriding records in a master plugin do not count toward this limit.

[DoubleYou]

A record is not necessarily a formid. I have tried to make it very clear that the limit is 2048 formids, not records. Also, as Greg says, overrides of previous masters utilize the master's formid, so they don't count toward the limit of 2048 formids. 

[Guba]

@DoubleYou
>A record is not necessarily a formid. I have tried to make it very clear that the limit is 2048 formids, not records.

Sorry Buddy, IMHO you're partially wrong.
You are on the right side when you say that FormIDs are not records.
But seems that the limit of FormIDs is 2048, because it seems to be 4096.
Let's analyze the code inside a script xEdit is shipped with.

The script I'm referring to is named Find ESP Plugins which could be turned into ESL.pas
Well, in the initial part of the code there are these lines:

{
  Find plugins which can be converted to ESL
}
unit FindPotentialESL;

const
  iESLMaxRecords = $800; // max possible new records in ESL
  iESLMaxFormID = $fff; // max allowed FormID number in ESL

There are two constants declared that are iESLMaxRecords = $800; that in decimal value is equal to 2048, and
  iESLMaxFormID = $fff; that in decimal value is equal to 4096.

Browsing down the code there are nice instructions that are:

    if not IsMaster(e) then
      Continue;

    if Signature(e) = 'CELL' then
      HasCell := True;

    // increase the number of new records found
    Inc(RecCount);

    // no need to check for more if we are already above the limit
    if RecCount > iESLMaxRecords then
      Break;

So first the program checks if the reference provided by the interface "e" is a new record - that, according to this logic, it means that is not referenced by any other master - and in this case is pointless to proceed in the analysis, so it skips the rest of the loop.
Second, the program checks if there are CELL values in the record interface. If I'm right, this is used determine that the plugin can't be converted, because Skyrim engine has a bug that prevents the ESL to keep its values consistent, because they possibly will be overwritten by other plugins.
Third, the program increments the number of records, that are referenced in the plugin declared masters (not referenced in masters are skipped at the "continue").
Then there is a check against the iESLMaxRecords (2048) and the cycle breaks if this limit is reached. After the end of the loop there is another check against the maximum record number that makes the program to exit the checkup procedure.
But I'd like to draw your attention on these lines, the last lines in the loop:

    // get raw FormID number
    fid := FormID(e) and $FFFFFF;

    // determine the max one
    if fid > RecMaxFormID then
      RecMaxFormID := fid;

If the value got into "fid" by the FormID interface is higher than the previous one its stores in the ceiling value the found value.
The the loop ends, and the first check is against the iESLMaxRecords (2048) it simply exits the ChecKForESL routine without any unnecessary notice.

  // too many new records, can't be ESL
  if RecCount > iESLMaxRecords then
    Exit;
 
The cool magick comes a bit later, in this piece of code:

  AddMessage(Name(f)); // it gests the Plugin name from the passed interface

  if RecMaxFormID <= iESLMaxFormID then // It checks if there are less than the maximum allowed (4096) FormIDs 
    AddMessage(#9'Can be turned into ESL by adding ESL flag in TES4 header')
  else // and, if not, it means that there are FormIDs that fall outside the boundaries declared in the header of the plugin interface
    AddMessage(#9'Can be turned into ESL by compacting FormIDs first, then adding ESL flag in TES4 header');
  //  that means that there are unreferenced FormIDs. Here we see a nice and neat way to find if there are unreferenced FormIDs  in the plugin structure.
  // check if plugin has new cell(s)
  if HasCELL then
    AddMessage(#9'Warning: Plugin has new CELL(s) which won''t work when turned into ESL and overridden by other mods due to the game bug');

Since we see this code, the thing to say is that there are TWO conditions to check if a plugin can be flagged as ESL:
 1. the number of records, that must be lower than 2048
2. the number of FormIDs, that must be lower than 4096.

Reading the pascal code that ships with xEdit was really informative!

[Guba]

If someone at STEP knows ElminsterAU he could ask him for an ultimate opinion on this topic....

Maybe....

[z929669]

I'm not familiar with that script, but I quickly looked through your post, so I'm assuming that your analysis is basically accurate.

Regardless, one cannot necessarily infer hard rules based on a scripted procedure. It depends on if the script was written to conform to the hard rules or if it was written to conservatively process with some padding or other calls made by the author.

So you could post over on AFKmods with your question, pointing them to this evidence. Then let us know.

[Greg]

When Bethesda initially created the ESL format, the general rule is that any FormIDs from $001 through $7FF are reserved for hard-coded records in the base game file and FormIds from $800 through $FFF are available for use. When Bethesda created Fallout 4, ESL plugins with Version 0.95 follow this same logic. The difference is that in some later release of Fallout 4, Bethesda changed the engine so that ESL plugins with Version 0.95 follow this same logic but ESL plugins with Version 1.00 can use FormIDs from $001 through $FFF and none of these are treated as injections into the base game.

Reference xEdit discord and search in:#mod-talk 7FF.

EDIT: I forgot to mention that $000 is excluded because this is apparently an invalid FormId.

[Guba]

@z929669

I'm not inferring any rule based on a script. Im making some possible deductions, and in fact I was asking to confirm my deductions.
What I can say is this: if the script is bugged, after such a long time xEdit is around, someone would have fixed.
This script is in the set which is shipped with xEdit since version 4.0 of SSEDIT. I have downloaded all the versions of SSEDIT available on Nexus to investigate. You can check this by yourself, the script is a PAS (pascal) file which is datestamped on 05/31/2018 3:39. The same file appears in the 4.0.4 release.
SO I refuse to think that, if the script were so grossly bugged, it hadn't been fixed before.
Such a severe bug, if existed in realty, had compromized so many gamesaves that it should have been investigated and fixed by the authos of SSEDIT that, believe me, are good programmers.
Also I can say that there is a lack of first-hand information about some Bethesda internals on functional mechanics of the Skyrim engine.
Sentences like "When Bethesda created Fallout 4, ESL plugins with Version 0.95 follow this same logic. " are without meaning if they can't be supported by any official (internal) documentation on functional specifications. Functional specs that are proprietary of the Bethesda lab. 
I was used to be an IT manager, and I know pretty well how things can be made in professional environments.
If you do not have standards you cannot work in team on large projects, and standards are documentend only on internal documents.
So every conjecture that can be made upon any kind of behavior that software may have does remain what it is: a conjecture. And should not be sold as a Gospels truth.
All that I can see around is derived by deductions of the modders, so the fact that I'm supposedly inferring hard rules based upon scripts is not so different from what has been done until now, that is observing the behavior of the software and deducting things from those behaviors.
One fact remains, and this is undisputable: the scripts are made in a way that contradict some of what is the common information that's around.
And a lot of misinformation and confusion is around too.
I say it again: an opinion of someone of the SSEDIT developer team should derime the problem.
By my side I'm going to download the SSEdit code and I will make a review by myself.
We shall see.

A little addendum to what I wrote before.
In the "Edit Scripts" folder of the 4.0.4 deliverable there are two copies of the same file with different names, but with equal content:
"Find ESP plugins which could be turned into ESL.pas"
"Find plugins which could be turned into ESL.pas"
Now I'm asking why there are tow idendical failes with different names. My opinion is to find the file with filter option that can include or not the word "ESP". But it is only an opinion.
Both files have the same CONST declaration, the file contents are identical.
So now someone can tell me why the same script has two times the same decalarations, if these are wrong?
 

Edited February 16, 2023 by Guba
typos

[z929669]

2 hours ago, Guba said:

@z929669

I'm not inferring any rule based on a script. Im making some possible deductions, and in fact I was asking to confirm my deductions.
What I can say is this: if the script is bugged, after such a long time xEdit is around, someone would have fixed.
This script is in the set which is shipped with xEdit since version 4.0 of SSEDIT. I have downloaded all the versions of SSEDIT available on Nexus to investigate. You can check this by yourself, the script is a PAS (pascal) file which is datestamped on 05/31/2018 3:39. The same file appears in the 4.0.4 release.
SO I refuse to think that, if the script were so grossly bugged, it hadn't been fixed before.
Such a severe bug, if existed in realty, had compromized so many gamesaves that it should have been investigated and fixed by the authos of SSEDIT that, believe me, are good programmers.
Also I can say that there is a lack of first-hand information about some Bethesda internals on functional mechanics of the Skyrim engine.
Sentences like "When Bethesda created Fallout 4, ESL plugins with Version 0.95 follow this same logic. " are without meaning if they can't be supported by any official (internal) documentation on functional specifications. Functional specs that are proprietary of the Bethesda lab. 
I was used to be an IT manager, and I know pretty well how things can be made in professional environments.
If you do not have standards you cannot work in team on large projects, and standards are documentend only on internal documents.
So every conjecture that can be made upon any kind of behavior that software may have does remain what it is: a conjecture. And should not be sold as a Gospels truth.
All that I can see around is derived by deductions of the modders, so the fact that I'm supposedly inferring hard rules based upon scripts is not so different from what has been done until now, that is observing the behavior of the software and deducting things from those behaviors.
One fact remains, and this is undisputable: the scripts are made in a way that contradict some of what is the common information that's around.
And a lot of misinformation and confusion is around too.
I say it again: an opinion of someone of the SSEDIT developer team should derime the problem.
By my side I'm going to download the SSEdit code and I will make a review by myself.
We shall see.

A little addendum to what I wrote before.
In the "Edit Scripts" folder of the 4.0.4 deliverable there are two copies of the same file with different names, but with equal content:
"Find ESP plugins which could be turned into ESL.pas"
"Find plugins which could be turned into ESL.pas"
Now I'm asking why there are tow idendical failes with different names. My opinion is to find the file with filter option that can include or not the word "ESP". But it is only an opinion.
Both files have the same CONST declaration, the file contents are identical.
So now someone can tell me why the same script has two times the same decalarations, if these are wrong?
 

Sorry if I added confusion. My hands are in quite a few pots these days, and I just didn't have the time or the inclination to sort out the script or it's logic to lend anything useful to your analysis and was merely asserting that inference based on a script should be cautionary. That's all. Elminster is a reliable authority though, so I should probably just have stayed out of this.

All we can do is infer, true, but @Mousetick DID take the time to work out the script's logic to lend to your analysis (see best answer up top ... I marked it, since it seems to address your original question best. You can confirm or deny by upvoting or downvoting). He's usually quite thorough and accurate in my XP, so I think you'll find his response useful.