TES5Edit scripting - Slow function wbCopyElementToFile

[MilletGtR]

Hi fellow modders, I'm MilletGtR, the creator of iActivate on the Nexus.

 

I've just recently gotten into the world of pascal scripting, so please forgive me if my quesions have obvious answers :) I'll get right into my issue.

 

In iActivate, I have altered some of the game setting strings to hide the Activate text, such as "Open", "Talk" and "Search". The problem there is that some items in the world of Skyrim have an "Activate Text Override" defined, meaning it overwrites the game setting (Open, Search, Talk and so on). Initially I had the ambition to edit all of these objects by hand, to be included in the mods .esp, but that took a stop when I realized that it would require patch after patch to be compatible with some of the more popular mods out there.

 

Instead I decided to learn Pascal and create a script in TES5Edit that people can run and make their own patch for their own load order. I've now created a fully functional script which finds all relevant records which contains the element 'RNAM' (which is the code for Activate Text Override), copies them to a new file, and at the same time removes the string attatched to 'RNAM'.

 

Now I've come to my issue: The script works very well on small .esp's, but the bigger the plugin file, the longer it takes per record copied. It seems to be exponential.

 

Down below is my complete script. Due to my research I suspect that the culprit is the wbCopyElementToFile function, which to me seems to scan the entire record every single time the procedure is run.

 

 

{
Creates a patch for iActivate that selects all relevant records containing the element 'RNAM' or
"Activate Text Override", and modifies them to remove the string that is defined in 'RNAM'.
}
unit UserScript;

var
iAFile: IInterface;
RNAMList: TStringList;
recs: array [0..50000] of IInterface;

function Initialize: integer;
var
    i: Integer;

begin
i := MessageDlg('Before running this script, you should be aware that:'+chr(13)+'1. You need to select all plugins that you wish to apply this script to, before running the script.'+chr(13)+'2. If you create this patch and later remove any of its master plugins you selected in step 1, you need delete the previously created patch, and create a new patch with this script, or the game will crash on startup.'+chr(13)+'3. This may take a while to process, so be patient.'+chr(13)+''+chr(13)+'Are you sure you wish to continue?', mtConfirmation, [mbYes, mbCancel], 0);
if i = mrYes then begin
    RNAMList := TStringList.Create;
    AddMessage('Building RNAM list, please stand by..');
end else begin
Result := 1;
Exit;
end;
end;

function Process(e: IInterface): integer;
var
ID: integer;
s: string;
begin
    //Ignores all signatures except the ones below.
if     (Signature(e) 'ACTI') and
    (Signature(e) 'CONT') and
    (Signature(e) 'FLOR') and
    (Signature(e) 'FURN') then
    Exit;
    //If the searched record does not contain the element 'RNAM', then skip record.
if not ElementExists(e, 'RNAM') then
Exit;
    
    //IntToHex converts the Value to a Hexadecimal string. IntToHex(Value, Digits) where Digits is the desired character length.
s := IntToHex(FormID(e), 8);
ID := RNAMList.IndexOf(s);
if ID = -1 then begin
recs[RNAMList.Count] := e;
RNAMList.Add(s);
    AddMessage('Copying ' + FullPath(e));

end else
recs[iD] := e;
end;

function Finalize: integer;
var
i: integer;
r, t: IInterface;
begin
if RNAMList.Count 0 then begin
    // Creates a new file where the above defined records will be stored.
    iAFile := AddNewFile;
if not Assigned(iAFile) then begin
AddMessage('Failed to create patch.');
Result := 1;
Exit;
end;
for i := 0 to RNAMList.Count - 1 do begin
r := recs;
    // Adds the current plugin as a master file.
AddRequiredElementMasters(GetFile®, iAFile, False);

    // copy CELL record to patch, parameters: record, file, AsNew, DeepCopy
    t := wbCopyElementToFile(r, iAFile, False, True);
    SetElementEditValues(t, 'RNAM', '');

    AddMessage('Copied ' + FullPath®);
end;
AddMessage(Format('Patch file created with %d RNAM records.', [RNAMList.Count]));
end else
AddMessage('Script found no elements containing RNAM.');
RNAMList.Free;
end;

end.

 

If anyone has any idea as to why the script runs so slowly on big files, please reply with any information. I'm at the end of the rope, and can't figure this one out.

 

I should also mention that my script runs very well and rapidly in the Procedure function, but extremely slowly on the Finalize function.

 

Millet

Edited May 23, 2015 by MilletGtR

[Mator]

Hi Millet.

 

There are a few things in your script that would cause obvious slowdowns:

1. AddMessage will always be a slowdown when used.  It forces a callback to the main application to update the GUI, which is slow.  To avoid this issue you have a few options:

- Don't log messages at all

- Use a form with its own TMemo component.  This will be marginally faster if the TMemo is shown, and much faster if it is hidden (see Merge Plugins or Mator Smash for an example of what I mean)

 

2. The Process(e: IInterface) function is going to process more records than you're actually interested in.  As you yourself have stated in your code, you're only interested in records with signatures matching 'ACTI', 'CONT', 'FLOR', or 'FURN'.  Instead of using the Process function to go through ALL the records the user selected (which is going to be slow), you can use element traversal in your finalize function to get to the EXACT records you want to process.  You can still use the Process function to get the files the user selected.

 

Pastebin for syntax highlighting: https://pastebin.com/wgQM5RaJ

 

 

 

unit UserScript;
 
uses mteFunctions;
 
var
  files, masters: TStringList;
  userFile: IInterface;
 
procedure FixRNAM(g: IInterface);
var
  i: integer;
  r: IInterface;
begin
  for i := 0 to Pred(ElementCount(g)) do begin
    r := ElementByIndex(g, i);
    if not ElementExists(r, 'RNAM') then
      continue;
    AddMessage('    Fixing '+Name(r));
    try
      r := wbCopyElementToFile(r, userFile, false, true);
      SetElementEditValues(r, 'RNAM', '');
    except
      on x : Exception do AddMessage('    Failed to fix record. '+x.Message);
    end;  
  end;
end;
 
function Initialize: Integer;
begin
  { ... }
  files := TStringList.Create;
  masters := TStringList.Create;
  ScriptProcessElements := [etFile];  // process function will only get the files the user selected
end;
 
function Process(e: IInterface): Integer;
begin
  if StrEndsWith(GetFileName(e), '.dat') then exit; // skip hardcoded
  files.AddObject(GetFileName(e), TObject(e));
  AddMastersToList(e, masters);
end;
 
function Finalize: Integer;
var
  i: integer;
  f, g: IInterface;
begin
  if files.Count = 0 then begin
    AddMessage('User selected no files!  Terminating.');
    files.Free;
    masters.Free;
    exit;
  end;
  userFile := FileSelect('Select a file below:');
  if not Assigned(userFile) then begin
    AddMessage('Failed to create patch.');
    files.Free;
    masters.Free;
    exit;
  end;
  AddMastersToFile(userFile, masters, true);
  for i := 0 to Pred(files.Count) do begin
    f := ObjectToElement(files.Objects[i]);
    AddMessage('Processing file: '+files[i]);
    // process ACTI record group
    g := GroupBySignature(f, 'ACTI');
    if Assigned(g) then begin
      AddMessage('  Processing Group: ACTI');
      FixRNAM(g);
    end;
    // process FLOR record group
    g := GroupBySignature(f, 'FLOR');
    if Assigned(g) then begin
      AddMessage('  Processing Group: FLOR');
      FixRNAM(g);
    end;
  end;
 
  // clean masters
  CleanMasters(userFile);
 
  // free memory
  files.Free;
  masters.Free;
end;

end.

 

 

 

Also, only ACTI and FLOR records have RNAM elements, so you don't need to process CONT or FURN.

 

I applied this script to Skyrim.esm and it completed:

[Apply Script done]  Processed Records: 1, Elapsed Time: 00:37

 

 

-Mator

 

EDIT: Applied to a full load order of 154 mods and it completed in 1:54.  Got some errors on certain FormIDs probably due to master issues.  Will see if I can figure out a fix.

EDIT 2: Fix applied.  I stand by never using AddRequiredElementMasters in my scripts.  The way I just implemented is clearly better because it doesn't cause FormID mapping errors.  It's also faster.  With the version of the script above I processed a full load order of 153 mods, including all bethesda files and DLC in 00:50.  Would be even faster if I omitted the AddMessage() calls or used a progress form instead.

EDIT 3: Final version of the script which doesn't process CONT/FURN because they don't have RNAM elements:

https://pastebin.com/wgQM5RaJ

Edited May 23, 2015 by Mator

[DoubleYou]

Thanks, MilletGtR, for posting! If you message z929669, TechAngel85, EssArrBee, or Vuud (myself) on the Nexus, we can give you a fancy Mod Author badge.

[MilletGtR]

Cool, I will do! Do you have any information, or know of anyone who is a wizard at scripting in TES5Edit?

[DoubleYou]

Sheson works wonders, and he's working on DynDoLOD a lot. Zilav and Mator stop in every once in while.

[sheson]

Remove the  AddMessage from the loop. Writing lots of messages takes a long time.

Can't say much about the wbCopyElementToFile command itself. That is inside xEdit.

 

 

Edit: you could "optimize" away the array by adding e to a TList and then getting them out later with ObjectToElement

 

unit userscript;
var
  l: TList;

function Initialize: integer;
begin
  l:= TList.Create;
end;

function Process(e: IInterface): integer;
begin
  if not ElementExists(e, 'RNAM') then
    Exit;
  // Add Element e to list
  l.Add(e);
end;

function Finalize: integer;
var
  i: integer;
  e: IInterface;
begin;
  for i := 0 to Pred(l.Count) do begin
    e := ObjectToElement(l[i]);
    // now do whatever with Element e
  end;
end;

Edited May 23, 2015 by sheson

[MilletGtR]

Thank you for replying (and for your awesome mods) sheson.

 

I'm not at a computer so I can't try your suggestion, but the AddMessage is also written in the Process procedure, and that one functions very quickly dispite that. I can't explain that, but I will try it. If you have any other suggestions, please feel free to add more input :)

[ThreeTen]

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Edited May 24, 2015 by ThreeTen

[Mator]

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Good call.  Wasn't thinking about that.

[MilletGtR]

Thank you all for your very valuable inputs! I never expected to get such good answers (from such awesome people nonetheless). I will start figuring things out immediately :D

[MilletGtR]

 

 

Hi Millet.

 

There are a few things in your script that would cause obvious slowdowns:

1. AddMessage will always be a slowdown when used.  It forces a callback to the main application to update the GUI, which is slow.  To avoid this issue you have a few options:

- Don't log messages at all

- Use a form with its own TMemo component.  This will be marginally faster if the TMemo is shown, and much faster if it is hidden (see Merge Plugins or Mator Smash for an example of what I mean)

 

2. The Process(e: IInterface) function is going to process more records than you're actually interested in.  As you yourself have stated in your code, you're only interested in records with signatures matching 'ACTI', 'CONT', 'FLOR', or 'FURN'.  Instead of using the Process function to go through ALL the records the user selected (which is going to be slow), you can use element traversal in your finalize function to get to the EXACT records you want to process.  You can still use the Process function to get the files the user selected.

 

Pastebin for syntax highlighting: https://pastebin.com/wgQM5RaJ

 

 

 

unit UserScript;
 
uses mteFunctions;
 
var
  files, masters: TStringList;
  userFile: IInterface;
 
procedure FixRNAM(g: IInterface);
var
  i: integer;
  r: IInterface;
begin
  for i := 0 to Pred(ElementCount(g)) do begin
    r := ElementByIndex(g, i);
    if not ElementExists(r, 'RNAM') then
      continue;
    AddMessage('    Fixing '+Name(r));
    try
      r := wbCopyElementToFile(r, userFile, false, true);
      SetElementEditValues(r, 'RNAM', '');
    except
      on x : Exception do AddMessage('    Failed to fix record. '+x.Message);
    end;  
  end;
end;
 
function Initialize: Integer;
begin
  { ... }
  files := TStringList.Create;
  masters := TStringList.Create;
  ScriptProcessElements := [etFile];  // process function will only get the files the user selected
end;
 
function Process(e: IInterface): Integer;
begin
  if StrEndsWith(GetFileName(e), '.dat') then exit; // skip hardcoded
  files.AddObject(GetFileName(e), TObject(e));
  AddMastersToList(e, masters);
end;
 
function Finalize: Integer;
var
  i: integer;
  f, g: IInterface;
begin
  if files.Count = 0 then begin
    AddMessage('User selected no files!  Terminating.');
    files.Free;
    masters.Free;
    exit;
  end;
  userFile := FileSelect('Select a file below:');
  if not Assigned(userFile) then begin
    AddMessage('Failed to create patch.');
    files.Free;
    masters.Free;
    exit;
  end;
  AddMastersToFile(userFile, masters, true);
  for i := 0 to Pred(files.Count) do begin
    f := ObjectToElement(files.Objects[i]);
    AddMessage('Processing file: '+files[i]);
    // process ACTI record group
    g := GroupBySignature(f, 'ACTI');
    if Assigned(g) then begin
      AddMessage('  Processing Group: ACTI');
      FixRNAM(g);
    end;
    // process FLOR record group
    g := GroupBySignature(f, 'FLOR');
    if Assigned(g) then begin
      AddMessage('  Processing Group: FLOR');
      FixRNAM(g);
    end;
  end;
 
  // clean masters
  CleanMasters(userFile);
 
  // free memory
  files.Free;
  masters.Free;
end;

end.

 

 

 

Also, only ACTI and FLOR records have RNAM elements, so you don't need to process CONT or FURN.

 

I applied this script to Skyrim.esm and it completed:

[Apply Script done]  Processed Records: 1, Elapsed Time: 00:37

 

 

-Mator

 

EDIT: Applied to a full load order of 154 mods and it completed in 1:54.  Got some errors on certain FormIDs probably due to master issues.  Will see if I can figure out a fix.

EDIT 2: Fix applied.  I stand by never using AddRequiredElementMasters in my scripts.  The way I just implemented is clearly better because it doesn't cause FormID mapping errors.  It's also faster.  With the version of the script above I processed a full load order of 153 mods, including all bethesda files and DLC in 00:50.  Would be even faster if I omitted the AddMessage() calls or used a progress form instead.

EDIT 3: Final version of the script which doesn't process CONT/FURN because they don't have RNAM elements:

https://pastebin.com/wgQM5RaJ

 

 

 

This is an absolutely incredible script. I can't believe how ineffective mine is compared to yours. When I upload the newest (and hopefully final) version of iActivate, I will be sure to give major credits to you Mator!

Concerning the CONT and FURN records, you are right there. I was under the impression that some mods (like Mörskom) added RNAM to these tags, but they were actually on the activator (ACTI), thank you for clearing that up.

 

Concerning the AddMessage logs, I feel like they need to be there since a lot of (maybe inexperienced) users will use this script through TES5Edit. If there are no message logs maybe they will think that maybe the script causes the program not to answer (which appears to happen when not logging anything).

 

The script runs perfectly and very rapidly. Skyrim.esm took 12 seconds to process. With my script I reckon it would take an hour at least (so bad) :D

 

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Thank you ThreeTen, for this information. I will use this instead of "for i := 0 to Pred(files.Count) do begin" for sure.

Edited May 24, 2015 by MilletGtR

[ThreeTen]

No problem,  as a final suggestion I would forgo the entire Process(e: IInterface) function and port it into initialize:

 

function Initialize: Integer;

var

 i := integer;

 e := IInterface;

begin

 { ... }

 files := TStringList.Create;

 masters := TStringList.Create;

 ScriptProcessElements := [etFile];  // process function will only get the files the user selected

 

 for int i := 0 to Pred(FileCount) do begin

   e := FileByIndex(i);

   if StrEndsWith(GetFileName(e), '.dat') then continue; // skip hardcoded

   files.AddObject(GetFileName(e), TObject(e));

   AddMastersToList(e, masters);

  end;

end;

 

As tes5edit is still relatively new to the "autopatch" scene. people will have a much harder time knowing what files to select, or even basic steps such as applying scripts and how to navigate the program as they are not used to it.  There is a good possibility that they will miss selecting a file.  Using this method will minimize that risk as it will go through all of their files no matter what.

Edited May 24, 2015 by ThreeTen

[MilletGtR]

Thank you for your suggestion ThreeTen, and while I respect it, I feel like someone who is inexperienced will have a big chance to make the mistake of removing one of the plugins' masters (any file in their load order), and will get instant CTD when entering skyrims main menu. I will post a guide along with this optional file, as well as on Nexus, to guide the inexperienced/unsure users.

[ThreeTen]

Hrm, I do not think you may understand what the final CleanMasters() function does.  Even though all of the masters are added to your patch at the beginning of script, when that function is called,  all masters that do not directly relate to your patch file gets removed.  Perhaps a final message that would tell them what files have become the new masters before quitting the patch would be useful in this case.

Edited May 25, 2015 by ThreeTen

[MilletGtR]

You are right in that I understand very little when it comes to scripting, seeing as I begun learning about a week ago :) Thank you for your proposal, I will probably implement your solution.

[ThreeTen]

Well considering that you wrote a functional script in a week is pretty damn impressive in and of itself.  I remember when I started learning tes5edit for my real shelter patcher and my first attempts were horrendous at best.  Mator cried after seeing it the first time.

 

Also I am pretty sure you know about

https://ck.uesp.net/wiki/TES5Edit_Scripting_Functions

But I wanted to let you know that you should log into the wiki when viewing it, as it seems all of the change/additions over the last few months are not viewable to anyone who is not logged in (which is ridiculously silly).

Edited May 25, 2015 by ThreeTen