Jump to content
  • 0
MilletGtR

TES5Edit scripting - Slow function wbCopyElementToFile

Question

Hi fellow modders, I'm MilletGtR, the creator of iActivate on the Nexus.

 

I've just recently gotten into the world of pascal scripting, so please forgive me if my quesions have obvious answers :) I'll get right into my issue.

 

In iActivate, I have altered some of the game setting strings to hide the Activate text, such as "Open", "Talk" and "Search". The problem there is that some items in the world of Skyrim have an "Activate Text Override" defined, meaning it overwrites the game setting (Open, Search, Talk and so on). Initially I had the ambition to edit all of these objects by hand, to be included in the mods .esp, but that took a stop when I realized that it would require patch after patch to be compatible with some of the more popular mods out there.

 

Instead I decided to learn Pascal and create a script in TES5Edit that people can run and make their own patch for their own load order. I've now created a fully functional script which finds all relevant records which contains the element 'RNAM' (which is the code for Activate Text Override), copies them to a new file, and at the same time removes the string attatched to 'RNAM'.

 

Now I've come to my issue: The script works very well on small .esp's, but the bigger the plugin file, the longer it takes per record copied. It seems to be exponential.

 

Down below is my complete script. Due to my research I suspect that the culprit is the wbCopyElementToFile function, which to me seems to scan the entire record every single time the procedure is run.

 

 

{
Creates a patch for iActivate that selects all relevant records containing the element 'RNAM' or
"Activate Text Override", and modifies them to remove the string that is defined in 'RNAM'.
}

unit UserScript;

var
iAFile: IInterface;
RNAMList: TStringList;
recs: array [0..50000] of IInterface;


function Initialize: integer;
var
    i: Integer;

begin
i := MessageDlg('Before running this script, you should be aware that:'+chr(13)+'1. You need to select all plugins that you wish to apply this script to, before running the script.'+chr(13)+'2. If you create this patch and later remove any of its master plugins you selected in step 1, you need delete the previously created patch, and create a new patch with this script, or the game will crash on startup.'+chr(13)+'3. This may take a while to process, so be patient.'+chr(13)+''+chr(13)+'Are you sure you wish to continue?', mtConfirmation, [mbYes, mbCancel], 0);
if i = mrYes then begin
    RNAMList := TStringList.Create;
    AddMessage('Building RNAM list, please stand by..');
end else begin
Result := 1;
Exit;
end;
end;


function Process(e: IInterface): integer;
var
ID: integer;
s: string;

begin
   
 //Ignores all signatures except the ones below.
if     (Signature(e) 'ACTI') and
    (Signature(e) 'CONT') and
    (Signature(e) 'FLOR') and
    (Signature(e) 'FURN') then
    Exit;
   
 //If the searched record does not contain the element 'RNAM', then skip record.
if not ElementExists(e, 'RNAM') then
Exit;
    
   
 //IntToHex converts the Value to a Hexadecimal string. IntToHex(Value, Digits) where Digits is the desired character length.
s := IntToHex(FormID(e), 8);
ID := RNAMList.IndexOf(s);
if ID = -1 then begin
recs[RNAMList.Count] := e;
RNAMList.Add(s);
    AddMessage('Copying ' + FullPath(e));

end else
recs[iD] := e;
end;


function Finalize: integer;
var
i: integer;
r, t: IInterface;

begin
if RNAMList.Count 0 then begin
   
 // Creates a new file where the above defined records will be stored.
    iAFile := AddNewFile;
if not Assigned(iAFile) then begin
AddMessage('Failed to create patch.');
Result := 1;
Exit;
end;
for i := 0 to RNAMList.Count - 1 do begin
r := recs;
    
// Adds the current plugin as a master file.
AddRequiredElementMasters(GetFile®, iAFile, False);

   
 // copy CELL record to patch, parameters: record, file, AsNew, DeepCopy
    t := wbCopyElementToFile(r, iAFile, False, True);
    SetElementEditValues(t, 'RNAM', '');

    AddMessage('Copied ' + FullPath®);
end;
AddMessage(Format('Patch file created with %d RNAM records.', [RNAMList.Count]));
end else
AddMessage('Script found no elements containing RNAM.');
RNAMList.Free;
end;

end.

 

If anyone has any idea as to why the script runs so slowly on big files, please reply with any information. I'm at the end of the rope, and can't figure this one out.

 

I should also mention that my script runs very well and rapidly in the Procedure function, but extremely slowly on the Finalize function.

 

Millet

Edited by MilletGtR

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

Hi Millet.

 

There are a few things in your script that would cause obvious slowdowns:

1. AddMessage will always be a slowdown when used.  It forces a callback to the main application to update the GUI, which is slow.  To avoid this issue you have a few options:

- Don't log messages at all

- Use a form with its own TMemo component.  This will be marginally faster if the TMemo is shown, and much faster if it is hidden (see Merge Plugins or Mator Smash for an example of what I mean)

 

2. The Process(e: IInterface) function is going to process more records than you're actually interested in.  As you yourself have stated in your code, you're only interested in records with signatures matching 'ACTI', 'CONT', 'FLOR', or 'FURN'.  Instead of using the Process function to go through ALL the records the user selected (which is going to be slow), you can use element traversal in your finalize function to get to the EXACT records you want to process.  You can still use the Process function to get the files the user selected.

 

Pastebin for syntax highlighting: https://pastebin.com/wgQM5RaJ

 

 

 

unit UserScript;
 
uses mteFunctions;
 
var
  files, masters: TStringList;
  userFile: IInterface;
 
procedure FixRNAM(g: IInterface);
var
  i: integer;
  r: IInterface;
begin
  for i := 0 to Pred(ElementCount(g)) do begin
    r := ElementByIndex(g, i);
    if not ElementExists(r, 'RNAM') then
      continue;
    AddMessage('    Fixing '+Name(r));
    try
      r := wbCopyElementToFile(r, userFile, false, true);
      SetElementEditValues(r, 'RNAM', '');
    except
      on x : Exception do AddMessage('    Failed to fix record. '+x.Message);
    end;  
  end;
end;
 
function Initialize: Integer;
begin
  { ... }
  files := TStringList.Create;
  masters := TStringList.Create;
  ScriptProcessElements := [etFile];  // process function will only get the files the user selected
end;
 
function Process(e: IInterface): Integer;
begin
  if StrEndsWith(GetFileName(e), '.dat') then exit; // skip hardcoded
  files.AddObject(GetFileName(e), TObject(e));
  AddMastersToList(e, masters);
end;
 
function Finalize: Integer;
var
  i: integer;
  f, g: IInterface;
begin
  if files.Count = 0 then begin
    AddMessage('User selected no files!  Terminating.');
    files.Free;
    masters.Free;
    exit;
  end;
  userFile := FileSelect('Select a file below:');
  if not Assigned(userFile) then begin
    AddMessage('Failed to create patch.');
    files.Free;
    masters.Free;
    exit;
  end;
  AddMastersToFile(userFile, masters, true);
  for i := 0 to Pred(files.Count) do begin
    f := ObjectToElement(files.Objects[i]);
    AddMessage('Processing file: '+files[i]);
    // process ACTI record group
    g := GroupBySignature(f, 'ACTI');
    if Assigned(g) then begin
      AddMessage('  Processing Group: ACTI');
      FixRNAM(g);
    end;
    // process FLOR record group
    g := GroupBySignature(f, 'FLOR');
    if Assigned(g) then begin
      AddMessage('  Processing Group: FLOR');
      FixRNAM(g);
    end;
  end;
 
  // clean masters
  CleanMasters(userFile);
 
  // free memory
  files.Free;
  masters.Free;
end;

end.

 

 

 

Also, only ACTI and FLOR records have RNAM elements, so you don't need to process CONT or FURN.

 

I applied this script to Skyrim.esm and it completed:

[Apply Script done]  Processed Records: 1, Elapsed Time: 00:37

 

 

-Mator

 

EDIT: Applied to a full load order of 154 mods and it completed in 1:54.  Got some errors on certain FormIDs probably due to master issues.  Will see if I can figure out a fix.

EDIT 2: Fix applied.  I stand by never using AddRequiredElementMasters in my scripts.  The way I just implemented is clearly better because it doesn't cause FormID mapping errors.  It's also faster.  With the version of the script above I processed a full load order of 153 mods, including all bethesda files and DLC in 00:50.  Would be even faster if I omitted the AddMessage() calls or used a progress form instead.

EDIT 3: Final version of the script which doesn't process CONT/FURN because they don't have RNAM elements:

https://pastebin.com/wgQM5RaJ

Edited by Mator

Share this post


Link to post
Share on other sites
  • 0

Cool, I will do! Do you have any information, or know of anyone who is a wizard at scripting in TES5Edit?

Share this post


Link to post
Share on other sites
  • 0

Remove the  AddMessage from the loop. Writing lots of messages takes a long time.

Can't say much about the wbCopyElementToFile command itself. That is inside xEdit.

 

 

Edit: you could "optimize" away the array by adding e to a TList and then getting them out later with ObjectToElement

 


unit userscript;
var
  l: TList;

function Initialize: integer;
begin
  l:= TList.Create;
end;

function Process(e: IInterface): integer;
begin
  if not ElementExists(e, 'RNAM') then
    Exit;
  // Add Element e to list
  l.Add(e);
end;

function Finalize: integer;
var
  i: integer;
  e: IInterface;
begin;
  for i := 0 to Pred(l.Count) do begin
    e := ObjectToElement(l[i]);
    // now do whatever with Element e
  end;
end;
Edited by sheson

Share this post


Link to post
Share on other sites
  • 0

Thank you for replying (and for your awesome mods) sheson.

 

I'm not at a computer so I can't try your suggestion, but the AddMessage is also written in the Process procedure, and that one functions very quickly dispite that. I can't explain that, but I will try it. If you have any other suggestions, please feel free to add more input :)

Share this post


Link to post
Share on other sites
  • 0

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Edited by ThreeTen
  • Upvote 1

Share this post


Link to post
Share on other sites
  • 0

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Good call.  Wasn't thinking about that.

Share this post


Link to post
Share on other sites
  • 0

Thank you all for your very valuable inputs! I never expected to get such good answers (from such awesome people nonetheless). I will start figuring things out immediately :D

Share this post


Link to post
Share on other sites
  • 0

 

 

Hi Millet.

 

There are a few things in your script that would cause obvious slowdowns:

1. AddMessage will always be a slowdown when used.  It forces a callback to the main application to update the GUI, which is slow.  To avoid this issue you have a few options:

- Don't log messages at all

- Use a form with its own TMemo component.  This will be marginally faster if the TMemo is shown, and much faster if it is hidden (see Merge Plugins or Mator Smash for an example of what I mean)

 

2. The Process(e: IInterface) function is going to process more records than you're actually interested in.  As you yourself have stated in your code, you're only interested in records with signatures matching 'ACTI', 'CONT', 'FLOR', or 'FURN'.  Instead of using the Process function to go through ALL the records the user selected (which is going to be slow), you can use element traversal in your finalize function to get to the EXACT records you want to process.  You can still use the Process function to get the files the user selected.

 

Pastebin for syntax highlighting: https://pastebin.com/wgQM5RaJ

 

 

 

unit UserScript;
 
uses mteFunctions;
 
var
  files, masters: TStringList;
  userFile: IInterface;
 
procedure FixRNAM(g: IInterface);
var
  i: integer;
  r: IInterface;
begin
  for i := 0 to Pred(ElementCount(g)) do begin
    r := ElementByIndex(g, i);
    if not ElementExists(r, 'RNAM') then
      continue;
    AddMessage('    Fixing '+Name(r));
    try
      r := wbCopyElementToFile(r, userFile, false, true);
      SetElementEditValues(r, 'RNAM', '');
    except
      on x : Exception do AddMessage('    Failed to fix record. '+x.Message);
    end;  
  end;
end;
 
function Initialize: Integer;
begin
  { ... }
  files := TStringList.Create;
  masters := TStringList.Create;
  ScriptProcessElements := [etFile];  // process function will only get the files the user selected
end;
 
function Process(e: IInterface): Integer;
begin
  if StrEndsWith(GetFileName(e), '.dat') then exit; // skip hardcoded
  files.AddObject(GetFileName(e), TObject(e));
  AddMastersToList(e, masters);
end;
 
function Finalize: Integer;
var
  i: integer;
  f, g: IInterface;
begin
  if files.Count = 0 then begin
    AddMessage('User selected no files!  Terminating.');
    files.Free;
    masters.Free;
    exit;
  end;
  userFile := FileSelect('Select a file below:');
  if not Assigned(userFile) then begin
    AddMessage('Failed to create patch.');
    files.Free;
    masters.Free;
    exit;
  end;
  AddMastersToFile(userFile, masters, true);
  for i := 0 to Pred(files.Count) do begin
    f := ObjectToElement(files.Objects[i]);
    AddMessage('Processing file: '+files[i]);
    // process ACTI record group
    g := GroupBySignature(f, 'ACTI');
    if Assigned(g) then begin
      AddMessage('  Processing Group: ACTI');
      FixRNAM(g);
    end;
    // process FLOR record group
    g := GroupBySignature(f, 'FLOR');
    if Assigned(g) then begin
      AddMessage('  Processing Group: FLOR');
      FixRNAM(g);
    end;
  end;
 
  // clean masters
  CleanMasters(userFile);
 
  // free memory
  files.Free;
  masters.Free;
end;

end.

 

 

 

Also, only ACTI and FLOR records have RNAM elements, so you don't need to process CONT or FURN.

 

I applied this script to Skyrim.esm and it completed:

[Apply Script done]  Processed Records: 1, Elapsed Time: 00:37

 

 

-Mator

 

EDIT: Applied to a full load order of 154 mods and it completed in 1:54.  Got some errors on certain FormIDs probably due to master issues.  Will see if I can figure out a fix.

EDIT 2: Fix applied.  I stand by never using AddRequiredElementMasters in my scripts.  The way I just implemented is clearly better because it doesn't cause FormID mapping errors.  It's also faster.  With the version of the script above I processed a full load order of 153 mods, including all bethesda files and DLC in 00:50.  Would be even faster if I omitted the AddMessage() calls or used a progress form instead.

EDIT 3: Final version of the script which doesn't process CONT/FURN because they don't have RNAM elements:

https://pastebin.com/wgQM5RaJ

 

 

 

This is an absolutely incredible script. I can't believe how ineffective mine is compared to yours. When I upload the newest (and hopefully final) version of iActivate, I will be sure to give major credits to you Mator!

Concerning the CONT and FURN records, you are right there. I was under the impression that some mods (like Mörskom) added RNAM to these tags, but they were actually on the activator (ACTI), thank you for clearing that up.

 

Concerning the AddMessage logs, I feel like they need to be there since a lot of (maybe inexperienced) users will use this script through TES5Edit. If there are no message logs maybe they will think that maybe the script causes the program not to answer (which appears to happen when not logging anything).

 

The script runs perfectly and very rapidly. Skyrim.esm took 12 seconds to process. With my script I reckon it would take an hour at least (so bad) :D

 

In Finalize()

I would switch the:  " for i := 0 to Pred(files.Count) do begin"  to "for i := Pred(files.Count) downto 0 do begin",  that way it will start with the file lowest in the users load order, and copy over any potential winning overrides first rather than their masters.

(Since if I remember correctly, any further calls to wbCopyElementToFile on the same element will be ignored, so we gotta get the winning overrides to copy over first).

Thank you ThreeTen, for this information. I will use this instead of "for i := 0 to Pred(files.Count) do begin" for sure.

Edited by MilletGtR

Share this post


Link to post
Share on other sites
  • 0

No problem,  as a final suggestion I would forgo the entire Process(e: IInterface) function and port it into initialize:

 

function Initialize: Integer;

var
 i := integer;
 e := IInterface;
begin
 { ... }
 files := TStringList.Create;
 masters := TStringList.Create;
 ScriptProcessElements := [etFile];  // process function will only get the files the user selected
 
 for int i := 0 to Pred(FileCount) do begin
   e := FileByIndex(i);
   if StrEndsWith(GetFileName(e), '.dat') then continue; // skip hardcoded
   files.AddObject(GetFileName(e), TObject(e));
   AddMastersToList(e, masters);
  end;
end;
 
As tes5edit is still relatively new to the "autopatch" scene. people will have a much harder time knowing what files to select, or even basic steps such as applying scripts and how to navigate the program as they are not used to it.  There is a good possibility that they will miss selecting a file.  Using this method will minimize that risk as it will go through all of their files no matter what.
Edited by ThreeTen

Share this post


Link to post
Share on other sites
  • 0

Thank you for your suggestion ThreeTen, and while I respect it, I feel like someone who is inexperienced will have a big chance to make the mistake of removing one of the plugins' masters (any file in their load order), and will get instant CTD when entering skyrims main menu. I will post a guide along with this optional file, as well as on Nexus, to guide the inexperienced/unsure users.

Share this post


Link to post
Share on other sites
  • 0

Hrm, I do not think you may understand what the final CleanMasters() function does.  Even though all of the masters are added to your patch at the beginning of script, when that function is called,  all masters that do not directly relate to your patch file gets removed.  Perhaps a final message that would tell them what files have become the new masters before quitting the patch would be useful in this case.

Edited by ThreeTen

Share this post


Link to post
Share on other sites
  • 0

You are right in that I understand very little when it comes to scripting, seeing as I begun learning about a week ago :) Thank you for your proposal, I will probably implement your solution.

Share this post


Link to post
Share on other sites
  • 0

Well considering that you wrote a functional script in a week is pretty damn impressive in and of itself.  I remember when I started learning tes5edit for my real shelter patcher and my first attempts were horrendous at best.  Mator cried after seeing it the first time.

 

Also I am pretty sure you know about

https://www.creationkit.com/TES5Edit_Scripting_Functions

But I wanted to let you know that you should log into the wiki when viewing it, as it seems all of the change/additions over the last few months are not viewable to anyone who is not logged in (which is ridiculously silly).

Edited by ThreeTen

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By packwatch
      Screenshot of the problem:

      I dragged SSEEdit from my 1080P Monitor to my 4K Monitor. Most things scaled properly, except for the font and title bar. I did it again. The font got smaller.
      My monkey brain thought this was funny, so I continued until I couldn't read it anymore. Well, there was NOTHING funny about it AT ALL.
      When I restarted, the font stayed the same.
      Overriding Windows DPI scaling to System for the Mod Organizer and xEdit applications didn't work. Running xEdit manually outside of MO didn't work. Changing theme didn't work. Deleting and reinstalling xEdit didn't work. Setting my windows scaling to 350% makes the font a little bigger, but still barely readable. Changing my Main Display in Windows/Nvidia and relaunching didn't work.
      I've been crying from troubleshooting an infinite loading screen error for the past 2 days and I need xEdit to help troubleshoot it. I miss Lydia. Please help.
    • By keithinhanoi
      Over time, and by reading comments of mod authors, I have learned that for certain record types displayed in TES5Edit, when using the conflict filter feature, some records are incorrectly identified as conflicting (ie., "conflicting" = overridden by a plugin which comes later in the load order.)
       
      For example, ripple, the author of Inconsequential NPCs has explained that location persistent references (LCPR sub-records in LCTN / Location records) supplied by different plugins are not actually overwritten by the last mod in the load order LCTN for certain locations (source). The implication here is that all those references are combined and used from all mods with that record type when Skyrim is started. So in other words, when making a compatibility patch in TES5Edit, you do not need to copy overrides for those particular records.
       
      I have asked in a number of places which records types do not need to be carried forward into compatibility patches, but have never received a reply, and still to this day have not found a definitive list in one place. Well, I'd like to change that, and I need your help, if this is something you are knowledgeable about.
       
      Below is a list of record types, grouped by category, that I have read comments saying they are incorrectly identified as conflicting, because they are actually combined at runtime:
       
      A List of Non-Conflicting Record Types seen in TES5Edit [WIP]
      Default Object Manager (DOBJ)Record sub-record types:
      DNAM - Objects  (Confirmed here) Dialogue Topic (DIAL) Record sub-record types:
      TFIC - Info Count  (Confirmed - sources: here & here) Dialogue Information (INFO) Record sub-record types:
      PNAM - Previous Info  (Confirmed here) Idle Animation (IDLE) Record sub-record types:
      ANAM - Related Idle Animations  (Confirmed here) Location (LCTN) Record sub-record types:  (Confirmed here)
      ACPR - Actor Cell Persistent Reference LCPR - Location Cell Persistent Reference RCUN - Reference Cell UNique ACSR - Actor Cell Static References LCSR - Location Cell Static Reference RCSR - Reference Cell Static Reference ACEC - Actor Cell Encounter Cell LCEC - Location Cell Encounter Cell RCEC - Reference Cell Encounter Cell ACID - Actor Cell Marker Reference LCID - Location Cell Marker Reference ACEP - Actor Cell Enable Point LCEP - Location Cell Enable Point NOTE: Other LCTN sub-record types require conflict management.
      (Confirmed - sources: here, here, here & here)

      Story Manager Quest Node (SMQN) Record sub-record types: (Confirmed - source: here & here)
      SNAM - Child sub-records QNAM - Quest Count / Quests Story Manager B??? Node (SMBN) Record sub-record types:
      SNAM - Child sub-records  (Confirmed here) For more details about how the above listed sub-record types merge at runtime, please see this excellent opening thread post by Arthmoor from 12 March 2014. Many thanks to him for confirming / explaining all of these, and a tip of the hat to MonoAccipitor for noticing Arthmoor's post.
       
      I will update this list with additional confirmed non-conflicting record types based on your replies.
      Thanks in advance for your help, and let's hope others can benefit from this list!
    • By TechAngel85
      Discussion topic:
      SKSE64 by The SKSE Team
      Wiki Link
       
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.