Current Skyrim Scripting Practices and Merging

[kryptopyr]

Mator, I hope I didn't come across as offended by your comments. Not only do I have a vast amount of respect for your thoughts on this matter, but I happen to agree with you, at least in theory, on nearly everything you've stated.  It's just that my experience in trying to work with other mod authors leaves me with very little expectation of seeing something like this get implemented within the current mod community. Despite my pessimism about the situation, I still think the discussion is one that is certainly worth having (though maybe we should move it to a new thread?).
 
So, that being said, here we go...

There's a huge difference in Mod Configuration Menus and the quest interface you're proposing. MCMs provide a direct benefit to the mod and mod author.  They're sleek, they're easy for users, and they're relatively simple to set up.  There is no real reliance on or need to cooperate with other mods (beyond SkyUI).  And SkyUI is one of the only mods massive enough to be able to set this sort of standard all on it's own. 
 
Setting up a quest interface for other mods to tie into provides a much more indirect benefit.  If no other mods use the system, then the mod implementing it doesn't gain anything, even indirectly.  The mod author still has to do the work, but any benefit they receive is dependent on other mods using the system and hooking into the their mod.  Trying to convince the majority of mod authors to implement special features within their mods when they may not even directly benefit from that effort is likely to be a losing battle.  I strongly believe that this is something the Unofficial patches should implement.  If you could convince the UP team to use this method, then I think the weight of USKP could well drive others to follow their approach.  There are few (if any) other mods that could carry enough weight to do this, so barring that you'd need to convince the authors of at least a dozen (imo) of the largest mods to cooperate.
 
Honestly, I see think this is about as likely as throwing a dozen cats into a swimming pool and expecting them to cooperate with one another in order to get themselves out.  Many modders are fairly cooperative, to a point, but they are also strongly opinionated and tend to prefer tried and true methods over new approaches...at least until someone else paves the way and proves the theory sound, and even then it can be a hard sell.
 

There is a HUGE difference between using Quest.GetQuest with a quest that exists for the intention of serving as an interface for other mods to tie into and using Game.GetFormFromFile or Game.GetFileByName to grab arbitrary information from arbitrary mods. In one hand the mod you're interfacing with with a component made for you to tie into, while in the other you're relying on arbitrary "random" things like filenames and FormIDs.

I agree with you 100%, but I was talking about using Quest.GetQuest for current existing mods (not theoretical mods that include interface quests specifically designed for communicating with other mods).  Using Quest.GetQuest for current mods would just be grabbing arbitrary information.  If the quest isn't designed specifically for mod compatibility, then the author could decide to change the quest just as easily as randomly deciding to change the mod's name.  And they would probably be even more likely do so, given that this isn't a standard way for existing mods to communicate with one another.  I don't think there are many modders who would really think twice about altering the name of a quest within their mod or aliases within that quest, and it wouldn't occur to many that the change could possibly create compatibility headaches for a mod using the Quest.GetQuest function.  I wasn't even aware of this function myself until very recently.
 

No, script "breaks" are FAR easier to fix. You change a few lines of code and everyone re-runs the script, and everything is fixed for everyone. If we're talking about patching hundreds of files, which is easier:
- Editing hundreds of ESP files
- Editing a few lines of code in a script

I think we may be talking about two different things here.  What I meant by 'breaking' a plugin patch is that if an author changes a record or formID then that will impact any esp patch that is referencing that record just as much as it would affect a mod using scripted compatibility via Game.GetFormFromFile.  Changing the record or FormID would break either form of compatibility in this case.  However, many users (and even modders) aren't generally in the habit of opening up the scripts within a mod and looking through them for this sort of potential conflict (and there's no way to easily cross-reference the scripts with the forms they're referencing to check for changes or conflicts).  Compare that to the growing number that are at least marginally comfortable with looking for problems and conflicts in TES5Edit, and it becomes much easier to identify and fix problems in plugin patches than it is in mods that use scripted compatibility.  
 
I'm not arguing against the value of TES5Edit scripts for creating certain mods or improving compatibility among mods.  Honestly, I think TES5Edit scripts are fantastic, and I wish I was more familiar with creating them (I'm slowly working on addressing that problem, thanks to you).  However, even seeing what amazing things you're capable of doing with scripts, I still can't imagine creating most of my most using automated scripts.  WAF and probably CCF could almost certain have been created much more efficiently using scripts, but the others?  It just depends on what the mod is trying to accomplish.  I definitely recognize that automated scripts are massively more efficient at what they do than manual edits could ever be.  But the more exceptions you have to deal with, and the more complex those exceptions are, the more quickly you start to eat away at that gain in efficiency.
 
 

This is extremely pessimistic. I strongly disagree with this paragraph and find it insulting. I have defined the standard of merging plugins. I have helped define the standard of xEdit scripts. I have built dozens of automated solutions that operate on standards. YES, standards are not "immutable", but that doesn't mean we can't operate on them. A world without standards is chaos.

I certainly did not intend it to be an insult, for you or anyone else.  I was simply trying to point out that despite the less-than-ideal compatibility methods that many modders currently use, those methods still tend to mostly function as intended, because they are generally seen and treated as standards within the community, even if they do happen to be flawed, delicate, or otherwise imperfect.  It isn't an ideal system, and I agree that the perceived standards are constantly changing.  However, even the most widely used community standards can't be considered truly universal.  Modders use the tools they're familiar with and the ones they feel work best for them within the setting of their own knowledge and their interaction with the broader community. 
 
The ideal system, as you pointed out, would be a truly concrete and explicit standard that nearly everyone would be familiar with and would use.  Such a system would greatly benefit the entire community.  But I believe the only realistic way to establish that sort of universal, unquestionable standard is to have it established by Bethesda.  Even SkyUI and it's MCM are not universally used by all mod authors and mod users. 
 
That statement isn't at all intended to denigrate your work or lessen your accomplishments, and I'm definitely not suggesting that we should only be using "official" standards.  My intention was quite the opposite.

 

"the unofficial patch team can name their files whatever they want. No one has any right to tell them otherwise, and it is absurd to think that anyone should. Filenames are not made with the intention of "creating an interface".

 Yes. Yes. And yes.  I agree with all of this.  I'm not telling them what to do; I'm not even arguing with them about it or suggesting to them that they change it.  I haven't made a single complaint to them concerning this; I only asked after their reasons.  As you correctly point out, it's their choice to make.  I have merely expressed my reasons for disagreeing with their decision, that doesn't mean that I don't fully respect their right to make that decision.  I've also stated that I'm engaging in this discussion more for the sake of the debate then because I feel strongly about this.
 
I also agree that filenames are not designed specifically with the intention of creating an interface, but both patch files and scripts use them in that way.  This is fairly common practice, so regardless of whether it's a good standard for compatibility, the reality exists that file names are used in this way.  I'm not disagreeing with you that we could design much better, more effective compatibility standards.  But we can't simply ignore the widespread use of the current existing standards just because theoretically there are better options.
 

Would you be happy if there was an easy two-click solution to change a mod's master from Unofficial Skyrim Patch.esp to USLEEP.esp? Because I can make that. Then mod users/authors will have the power to change that at any time, effectively providing us with the advantages of the filename staying the same while not creating the mess that would be associated with it staying the same.

 Well, I'm already pretty happy.  Honestly.  I'm truly thrilled that they're moving to a unified patch, and there's not an ounce of sarcasm in that statement.  But, yes, I think your two-click solution does sound nice.
 

Also, I can't help but continue wondering how many mods are actually master-dependent on the USKP. (or otherwise dependent on a file with any filename)

I don't think it's all that many.  I think Arthmoor had some number that he gave last year(?) when they first proposed the idea of merging the unofficial patches.  Of course, I suspect that number only takes into account mods that are directly dependent and not the ones that are using scripts for silent compatibility.  But even if you include the scripted mods, it's a fairly small number.
 

I think it's a fun discussion too. Apologies if I get a bit defensive in my post. It's just how I do discussions like this.

Good, me too.  I'm enjoying the discussion, and I think the issue of compatibility standards is definitely a conversation worth having.  I apologize if I'm coming across as overly pessimistic or aggressive in my arguments.  To some degree, I feel a bit like I'm taking up the role of devil's advocate here, not because I don't agree with what I've stated, but because I do recognize that some of the counter arguments are equally strong and just as valid.  And as I stated before, I'm arguing most of these points much more strongly than I truly feel, since I do think the debate is an interesting one.

 

I'm a big proponent of better mod communication.  In fact, it's something I feel rather strongly about, and it's an area where I feel the community as a whole would benefit greatly from even relatively small improvements in the current standards.   But I've chased this rabbit before.  Cross mod compatibility is something I've spent a lot of time on and have actually invested countless hours sending private messages to some of the major mod authors to try to establish a framework for a more efficient system of cross mod communication.   With several notable exceptions, I've been met with either unshakable skepticism or simple disinterest. 

 

For some authors, I think they just don't see enough direct personal gain to be worth the effort.  They are happy with their mods as things currently stand and feel that compatibility patches and Game.GetFormFromFile are perfectly sufficient to address any issues between mods.  Other authors are skeptical about any new approach to compatibility, regardless of whether or not it could directly benefit them.  This group would probably be willing to adapt if a new standard were established.  The problem is convincing enough major authors to take the first step and implement the new approach in order to build up the recognition needed to convince the rest.

 

So, yeah, if I seem somewhat pessimistic about this idea, it's because I feel I've been quietly fighting that battle for several years and making very little progress.  I'm not against your idea of interface quests.  Actually, I think it's brilliant.

 

If you'll help me figure out the best way to implement an interface quest in some of my mods, then I'll be happy to do so.  I just don't know how to get other modders to adopt it.  I can contact the authors who have been receptive to my ideas of cooperation in the past and see if they'll consider using it as well, but I still believe that we'd need other major mod authors to sign on before this would get much notice.  I believe it's a worthy goal though.

[hishutup]

I moved several posts from here to this topic because it was no longer on topic and deserved it own thread.

Please do not get offended, if you feel your post is better suited in the other topic please inform one of the moderators.

Thank you.

[kryptopyr]

"forced to use USKP as a master because of a single cookpot record that it adds". Uhhh... that doesn't make any sense. You are not forced to use a mod as a master unless you want to modify content from that mod or make use of content it adds. If you want to make a mod compatible with the USKP you make sure your changes in override records don't conflict with theirs, and that's it. The USKP doesn't add many "new object references" to the game, and if your mod does happen to conflict with one of those you need a compatibility patch, not a master dependency on your main mod.

 

I chose this example because, if I remember correctly, this is a new record that does cause issues with mods that change Proudspire.  Thank the gods that the USKP doesn't add any more new objects than it does.  However, even if those new records are relatively few in number, they can create disproportionate headaches for certain mods.  And it's hard not to feel forced to deal with the issue given that it's the unofficial patch, and if you don't address it, then the majority of your users are going to experience that conflict. 

 

I've had to deal with this type of situation, where new records added by USKP were conflicting with changes from my mod, in at least three or four of my mods, so I don't consider this problem quite as rare and unusual as you seem to. 

 

Do you know how much crap I'd get from users if I released a house mod and had a separate compatibility patch required for using it with the USKP?  I can already hear the comments.  "OMG Why are there 2 esps!?"  "Do I really need the patch?"  "What's the second file for?"  "Everyone uses the Unofficial patches, why can't you make this into a single file."  "Why should I install two plugins just for a single house?  No thanks, I'll pass."  And on, and on, and on.   In the end, most modders in this situation will either make the USKP a master dependency or use Game.GetFormFromFile to edit the offending record via script. 

 

 

Edit: found the thread that referenced this issue.

As it turns out, the record is added by UHFP, not USKP, so I'll concede that it wasn't a perfect example.  Regardless, I feel that the point itself is still valid.

 

Edit2:  @Mator, when you first described the idea of using an interface quest to reference specific records, my first thought was toward the new records added by the unofficial patches and how much easier it would be to deal with these conflicts if the UP had such as system.  If there is any set of mods that should be leading by example and setting the standard for best cross-mod compatibility practices, then it should be the unofficial patches.  Have you ever approached them with the idea?

Disclaimer: I'm not suggesting that we have any right to tell UP what to do or that this would be anything other than their choice.  I am merely stating my personal opinion that they would be the ideal candidates for your quest system.

[hishutup]

Hopefully this is the last post I need to move, sorry kyrpt.

[kryptopyr]

Hopefully this is the last post I need to move, sorry kyrpt.

Sorry, didn't mean to post again on the other thread after you had moved, but I guess you moved the previous posts while I was in the process of writing that last one.  Thanks for moving this discussion to it's own thread. 

[kryptopyr]

Arthmoor's comment on UP Dependencies:  https://www.afkmods.com/index.php?/topic/4004-unofficial-patches-legendary-edition-or-separate-files/?p=152037

 

This is interesting:

The list of USKP dependencies is irrelevent because the filename for that would not be changing. Which would eliminate issues with 62 files according to the Nexus data.

Disclaimer: I am not suggesting the UP team does not have the right to change their mind on this. Just pointing out that the name change doesn't seem to have been the original intention.

 

Edit:  I just figured out how to check for dependencies on nexus.  "Files that require this file" are actually listed under a mod's "Required" tab.  I never noticed or paid any attention to that before.

 

USKP =  ~110 files (I didn't count all of these since some are patches for a mod that also requires USKP or are translationed versions of USKP or are the DLC patches, etc.)

[Kesta]

I strongly believe that this is something the Unofficial patches should implement.  If you could convince the UPteam to use this method, then I think the weight of USKP could well drive others to follow their approach.

 

 

While the USKP would probably be the only mod big enough to implement something like this (like SkyUI did with MCM), I can already say it's not going to happen. UP's team priority is to fix bugs. In short, they're develloping the USKP as if it was for people who play vanilla + USKP only (plus de list of "recommanded mod in addition to the USKP" hosted on afkmods). Though they do know that most people mod they're game, and thus avoid drastic changes that would cause major compatibility issues, implementing such a framework is completely out of the scope of the project.

 

 

Edit : typo... 

Edited July 28, 2015 by Kesta

[Mator]

Mator, I hope I didn't come across as offended by your comments. Not only do I have a vast amount of respect for your thoughts on this matter, but I happen to agree with you, at least in theory, on nearly everything you've stated. It's just that my experience in trying to work with other mod authors leaves me with very little expectation of seeing something like this get implemented within the current mod community. Despite my pessimism about the situation, I still think the discussion is one that is certainly worth having (though maybe we should move it to a new thread?).

No I don't think you did. I just get a little defensive about my ideas sometimes. Sometimes unnecessarily.

 

 

So, that being said, here we go...

 

There's a huge difference in Mod Configuration Menus and the quest interface you're proposing. MCMs provide a direct benefit to the mod and mod author. They're sleek, they're easy for users, and they're relatively simple to set up. There is no real reliance on or need to cooperate with other mods (beyond SkyUI). And SkyUI is one of the only mods massive enough to be able to set this sort of standard all on it's own.

This is true, but I think the entire community can acknowledge that making mods interface with each other is a pain. I don't think it would be resisted. Given some nice MCM-like documentation, some mods setting an example, and some good PR, it could be adopted reasonably quickly. The best thing about it? It doesn't make Game.GetFormFromFile not work, it just allows mod authors who use that to switch to a better method if they so wish. It would be a smooth, easy transition.

 

 

Setting up a quest interface for other mods to tie into provides a much more indirect benefit. If no other mods use the system, then the mod implementing it doesn't gain anything, even indirectly. The mod author still has to do the work, but any benefit they receive is dependent on other mods using the system and hooking into the their mod. Trying to convince the majority of mod authors to implement special features within their mods when they may not even directly benefit from that effort is likely to be a losing battle. I strongly believe that this is something the Unofficial patches should implement. If you could convince the UP team to use this method, then I think the weight of USKP could well drive others to follow their approach. There are few (if any) other mods that could carry enough weight to do this, so barring that you'd need to convince the authors of at least a dozen (imo) of the largest mods to cooperate.

I agree we should see if we can convince Arthmoor to implement this type of interface.

 

 

I agree with you 100%, but I was talking about using Quest.GetQuest for current existing mods (not theoretical mods that include interface quests specifically designed for communicating with other mods). Using Quest.GetQuest for current mods would just be grabbing arbitrary information. If the quest isn't designed specifically for mod compatibility, then the author could decide to change the quest just as easily as randomly deciding to change the mod's name. And they would probably be even more likely do so, given that this isn't a standard way for existing mods to communicate with one another. I don't think there are many modders who would really think twice about altering the name of a quest within their mod or aliases within that quest, and it wouldn't occur to many that the change could possibly create compatibility headaches for a mod using the Quest.GetQuest function. I wasn't even aware of this function myself until very recently.

I would not recommend using Quest.GetQuest unless it was with an interface quest. That's just another rabbit hole.

 

 

I think we may be talking about two different things here. What I meant by 'breaking' a plugin patch is that if an author changes a record or formID then that will impact any esp patch that is referencing that record just as much as it would affect a mod using scripted compatibility via Game.GetFormFromFile. Changing the record or FormID would break either form of compatibility in this case. However, many users (and even modders) aren't generally in the habit of opening up the scripts within a mod and looking through them for this sort of potential conflict (and there's no way to easily cross-reference the scripts with the forms they're referencing to check for changes or conflicts). Compare that to the growing number that are at least marginally comfortable with looking for problems and conflicts in TES5Edit, and it becomes much easier to identify and fix problems in plugin patches than it is in mods that use scripted compatibility.

Yes, I agree. I was talking about xEdit scripts. I don't really like Papyrus-based compatibility, to be honest, but I would find it more reasonable with the Interface Quest design pattern. My responses to your next two quotes are directed largely at the subject of general compatibility approaches and xEdit scripting.

 

 

I'm not arguing against the value of TES5Edit scripts for creating certain mods or improving compatibility among mods. Honestly, I think TES5Edit scripts are fantastic, and I wish I was more familiar with creating them (I'm slowly working on addressing that problem, thanks to you). However, even seeing what amazing things you're capable of doing with scripts, I still can't imagine creating most of my mods using automated scripts. WAF and probably CCF could almost certain have been created much more efficiently using scripts, but the others? It just depends on what the mod is trying to accomplish. I definitely recognize that automated scripts are massively more efficient at what they do than manual edits could ever be. But the more exceptions you have to deal with, and the more complex those exceptions are, the more quickly you start to eat away at that gain in efficiency.

The gain in efficiency is most clear when you have a single script that applies to multiple mods/use cases. E.g. the Armor Mod Builder. I do acknowledge that not every mod is perfectly "automatable", because some do a lot of arbitrary things that can't be generalized upwards. That said, you can actually use some fairly simple and maintainable methods to generalize exceptions. Generally speaking, people are always capable of finding new or different ways of programming things that get better and more generic. There is a huge well of untapped potential here.

 

 

I certainly did not intend it to be an insult, for you or anyone else. I was simply trying to point out that despite the less-than-ideal compatibility methods that many modders currently use, those methods still tend to mostly function as intended, because they are generally seen and treated as standards within the community, even if they do happen to be flawed, delicate, or otherwise imperfect. It isn't an ideal system, and I agree that the perceived standards are constantly changing. However, even the most widely used community standards can't be considered truly universal. Modders use the tools they're familiar with and the ones they feel work best for them within the setting of their own knowledge and their interaction with the broader community.

Absolutely, it does work, but it could be better. My place in this community (I feel) is to build tools and standards that allow us to do what we're already doing, but do it faster, more efficiently, and better. Instead of the community spending hundreds of thousands of hours making tape-on-patches for a game collapsing under the weight of mods slapped on with glue and staples, we should renovate our approaches - the system of mod creation and application - so components don't conflict quite so often. The current approach to modding is a conflict nightmare. With more generic, automated approaches we will be able to do more with less.

 

 

Well, I'm already pretty happy. Honestly. I'm truly thrilled that they're moving to a unified patch, and there's not an ounce of sarcasm in that statement. But, yes, I think your two-click solution does sound nice.

I'll make it then, should be easy.

 

 

I'm a big proponent of better mod communication. In fact, it's something I feel rather strongly about, and it's an area where I feel the community as a whole would benefit greatly from even relatively small improvements in the current standards. But I've chased this rabbit before. Cross mod compatibility is something I've spent a lot of time on and have actually invested countless hours sending private messages to some of the major mod authors to try to establish a framework for a more efficient system of cross mod communication. With several notable exceptions, I've been met with either unshakable skepticism or simple disinterest.

 

For some authors, I think they just don't see enough direct personal gain to be worth the effort. They are happy with their mods as things currently stand and feel that compatibility patches and Game.GetFormFromFile are perfectly sufficient to address any issues between mods. Other authors are skeptical about any new approach to compatibility, regardless of whether or not it could directly benefit them. This group would probably be willing to adapt if a new standard were established. The problem is convincing enough major authors to take the first step and implement the new approach in order to build up the recognition needed to convince the rest.

 

So, yeah, if I seem somewhat pessimistic about this idea, it's because I feel I've been quietly fighting that battle for several years and making very little progress. I'm not against your idea of interface quests. Actually, I think it's brilliant.

Well, we should start dialing this in then, get some proof of concepts going, and start building a nice presentation. In a way, this is like selling a product. Ultimately, I think the determining factor is going to be mod authors pressuring each other to follow this "better standard" we'll be making. In order to make this happen we'll have to make a really clear, clean presentation on how this new system will work and what its advantages are. If we can educate a few authors to be able to "sell" this idea to other authors, everything else will fall into place. The first step is always the hardest.

 

 

If you'll help me figure out the best way to implement an interface quest in some of my mods, then I'll be happy to do so. I just don't know how to get other modders to adopt it. I can contact the authors who have been receptive to my ideas of cooperation in the past and see if they'll consider using it as well, but I still believe that we'd need other major mod authors to sign on before this would get much notice. I believe it's a worthy goal though.

Let's do it.

 

 

I chose this example because, if I remember correctly, this is a new record that does cause issues with mods that change Proudspire. Thank the gods that the USKP doesn't add any more new objects than it does. However, even if those new records are relatively few in number, they can create disproportionate headaches for certain mods. And it's hard not to feel forced to deal with the issue given that it's the unofficial patch, and if you don't address it, then the majority of your users are going to experience that conflict.

 

I've had to deal with this type of situation, where new records added by USKP were conflicting with changes from my mod, in at least three or four of my mods, so I don't consider this problem quite as rare and unusual as you seem to.

 

Do you know how much crap I'd get from users if I released a house mod and had a separate compatibility patch required for using it with the USKP? I can already hear the comments. "OMG Why are there 2 esps!?" "Do I really need the patch?" "What's the second file for?" "Everyone uses the Unofficial patches, why can't you make this into a single file." "Why should I install two plugins just for a single house? No thanks, I'll pass." And on, and on, and on. In the end, most modders in this situation will either make the USKP a master dependency or use Game.GetFormFromFile to edit the offending record via script.

 

Edit: found the thread that referenced this issue.

As it turns out, the record is added by UHFP, not USKP, so I'll concede that it wasn't a perfect example. Regardless, I feel that the point itself is still valid.

Well at least it's not quite as bad as you thought, but I still will acknowledge your point here.

 

 

Edit2: @Mator, when you first described the idea of using an interface quest to reference specific records, my first thought was toward the new records added by the unofficial patches and how much easier it would be to deal with these conflicts if the UP had such as system. If there is any set of mods that should be leading by example and setting the standard for best cross-mod compatibility practices, then it should be the unofficial patches. Have you ever approached them with the idea?

Disclaimer: I'm not suggesting that we have any right to tell UP what to do or that this would be anything other than their choice. I am merely stating my personal opinion that they would be the ideal candidates for your quest system.

I think before we contact anyone we need to make a proof of concept. We need to quantify to ourselves how much is possible and how it's going to work. Once we've done that we can approach other authors with what we've found and a clear concise "sales pitch", so to say.

Edited July 29, 2015 by Mator