6ImpossibleThings

Thank you! AFAIK, our company has its own reasons to become friends with modding community (among other things, our engine-under-development should include explicit support for modding, including running mods on our servers (!)). So, when SUMMON - which was indeed started as a hobby project - got to where it is now, we raised this question with the management, and they agreed to spend some resources on making a (free and open-source) product out of it.

Wait. Let's keep in mind that Nexus is not the only mod source out there. By far the largest - yes, the only - nope. To set things up, even STEP (which I remember saying itself it is only the beginning of the modding journey, cannot find it now for some reason) has to refer to several other places - IIRC, at least SKSE and ENB DLLs are not on Nexus, and there is also - drumroll - AFK, where A.....r has a very long well-documented history of taking out older versions of his mods (hey, IIRC his exodus to AFK was triggered because of Nexus introducing this very retaining policy). And then, there is mod site #2 out there - LL, which AFAIK has no retaining policy whatsoever. Which means that my valid assumptions are moot only if we're assuming that there is nothing but Nexus out there, which is not exactly correct. With all due respect to Nexus (they are doing a surprisingly good job given their de-facto almost-monopoly stance), IMO relying on one single entity is not a good thing for community (see also https://en.wikipedia.org/wiki/Enshittification , which we all observe way too often, sigh); and SUMMON is designed to support pluginS (in plural) for different "file sources" - while now SUMMON has only Nexus plugin, we want to add at least LL plugin too, as well as to support singular-but-all-important-downloads such as SKSE and ENB DLLs (ENB versioning is a big pain in the neck, BTW). and let's also not forget about the improvements which are brought to the table by auto-upgrading to a new version (and with zero maintenance costs for the guide maintainer!) - which does stand most of the time, and which is fundamentally unavailable when referring by hash (indeed, as you noted, what exactly the hash is based on, is immaterial). Bottom line: yes, there ARE cases out there when referring by hash is inevitable . However, most of the time referring by "take latest version of the mod" is a better choice. As a result, for SUMMON we want to refer to the latest-greatest version by default, while providing an option (as "preferred hash" I described) to deal with of course, this whole thing is not carved in stone, we'll know MUCH more as we go ahead... Which is also consistent with the dual nature of the references. Stay tuned, and thanks! I learned a thing or three about vision of you folks. :-)

@z929669 Thanks for sharing your thoughts on this matter. :-) > I was just asking if SUMMON used AI to deduce EVERYTHING from the text of an existing guide ... Certainly not. > or if the first step is to produce a completed, validated, error-free build and then use SUMMON to deduce and 'record' json instructions from the result (files/folders in the VFS or mod manager or whatever) which can then be reused by someone wanting the same build. Not exactly this either. Unlike WJ and Nexus Collections, SUMMON does NOT require build to be anywhere complete. Install one mod using guide (EXACTLY as you would do it without SUMMON) - run deduction - you already got a modpack (="automated guide") for this one mod. Install another 5 mods - run deduce - now you got "automated guide"=modpack for 6 mods. So, the build for SUMMON does NOT have to be "completed", NEITHER "validated", NEITHER "error-free". SUMMON modpack (unlike WJ/NC) is merely a formalized and automated step-by-step guide, and if SUMMON modpack is based on a text guide - SUMMON modpack simply inherits all the problems (or lack of the problems) from the text guide. If the guide is rock-solid - resulting SUMMON will be rock-solid too, if not - well, tough luck, we cannot possibly do any magic here. Or, from a bit different perspective - unlike WJ/NC, SUMMON is modular (and text-based too) - so it is easy to combine things; in fact, SUMMON is all about collaboration and building upon work of the others. > Have you investigated Nexus Collections? They are Nexus' answer to Wabbajack and seem very similar to SUMMON Thanks for pointing them out - I didn't know about them. However, I took a very quick look at them and they seem to be similar to WJ, and not to SUMMON. They seem to be binary (="you don't see what has changed", which is a Big Problem(tm)), they seem monolithic not modular (even a bigger problem), and I expect them to have exactly the same problems as WJ with upgrading (more on upgrading below). All in all, in all the aspects I can see, NC seems to be similar to WJ, NOT to SUMMON. > One of the tedious jobs of maintaining our guides is updating mod instructions, which often include significant changes to FOMOD instructions (see Embers XD update history as a case in point ...and these are just the changes we've made with respect to our 2.3 SSE guide). That mod has had tens or possibly hundreds of significant updates over the past couple years, such that previous instructions would be nonsensical to those acquiring the currently-available file(s). Collections get around this by referencing the archived file version, which never changes when the mod is updated (whether the mod page is unpublished or the file is archived/hidden for some reason). Please correct me if I'm wrong, but you seem to imply that referring by hash is a good thing; however, having a bit of experience with WJ, I am sure it is NOT. The problem with referencing by hash is that if the exact version of the file you're referring by hash, disappears/gets unpublished (which happens all the time in RL) - it is the end of the road, with WJ modlist going into "maintenance mode" until the author fixes the problem (producing that "completed, validated, and error-free" modlist), and re-publishes this huge binary. If you take a look at WJ - you'll see tons of modlists which are in "maintenance mode" (which is a fancy way of saying "sorry, you cannot use it, with ETA being unknown, maybe never"). Why? Exactly because of referring by hash. In contrast, text guides (such as STEP) and SUMMON mod packs are much more resilient to trivial changes. Sure, LARGE changes (such as those you mentioned) will still need to be handled manually (and SUMMON will provide a way to specify a preferred hash, more on it below) - but small and trivial changes are MUCH more frequent then large ones - and being unable to install just because an exact file disappeared - is a Bad Thing(tm). Just think about it: how often it happens that file is upgraded - but guide stays the same? I'd say it is like 95%+ of the time, and if an old file gets unpublished under these circumstances - it means that hash-oriented WJ becomes unusable, while STEP (and by extension, SUMMON) stands without even noticing the change (and probably even benefiting from the improvements in the upgraded mod - with zero maintenance from the guide side). Sure, SUMMON will be able to refer by hash - but IMNSHO, making guides which are requiring EXACT versions of the underlying mods is akin to writing software which requires EXACT versions of the software libraries - while it happens, it tends to create MUCH more problems in the long run then it solves (BTW, such practice is frowned upon in software community, and for a reason too). So, in the ideal world (a.k.a. SUMMON ;-)) , we see it as follows: - for each archive in JSON, we allow to specify BOTH "preferred hash" AND more generic "how to find the last version of the file". If file with "preferred hash" is found - great, it is guaranteed that it can be installed using exactly the instructions specified. However, if such file CANNOT be found - it is NOT the end of the road, and we can try getting a newer version - with like 95% chance that the same instructions will still work over a new file . Having had some experience with WJ, I feel it is a HUGE advantage of STEP (and of SUMMON) over WJ (and most likely, over NC too). > I assume the 'plugins' are the json scripts for mod-installation instructions. Not really. There is this sixitbb/summon project (available by the link in my signature), with .py files. Within summon folder, there is summonmm/plugins - it is Python language plugins which are handling specific things (such as plugins/globaltools/bodyslide.py handling, well, BodySlide). JSON scripts (a.k.a. "modpacks" = "automated instructions") are very different - they're produced by summon (which is aided by plugins), and then run by summon (again, aided by plugins). Just one example from the above. In a RL setup, we found that there is a case where we were able to deduce most of the install of "project rainforest", but it had 2 esps moved post-install (by MO2 "optional" window). Ok, we wrote a simple plugins/modtools/optional.py - which handles moving .esp's to/from 'optional' folder, which allowed us to get "clean" (="100% deduced") JSON for "project rainforest" (as can be seen in the JSON I posted earlier). And as you can imagine - there are TONS of such micro-tools and practices => which means we need TONS of plugins. And plugins for serious tools such as BodySlide are MUCH more complicated... > I was alluding to the "open source" concept here and wondering if the code base of the actual application (in addition to the plugins) would be publicly available. > Given you work for an Irish company, the app may be proprietary, but the development of instruction sets will be open? I'm just wondering if SUMMON is merely an open-source component of an otherwise proprietary application or if the project is entirely open source. Just curious. Sure, it will be available (whatever we have, it already is on GitHub), and it is entirely open source, free to use, and so on. The only reservation is that - except for external pull requests - actual development will be probably within an internal company source control, with only meaningful milestones pushed to GitHub. > Anyway, I find it interesting on a personal level and am following the project. Whether we use it in the future or not is an open question, but we're not really looking for an automated solution. Anyone is free to use it for installing Step builds just as they can do now with Wabbajack and Collections. We just haven't curated our builds for use with either of these tools, because it's a lot of additional work and maintenance for us. Of course, no problem; it is important to know that you don't object to creation of such automated guides based on STEP.

@z929669 To give a taste of it: here are some (deduced from real setup by SUMMON) JSON instructions: { "name": "comap", "installers": [ { "h": "BjI++vJgzuW/7AgWT9XrJJu6Da4VrmJ5Zho6A1WVXzM", "type": "FOMOD", "params": { "sel": [ { "step": "Core Files", "grp": "CoMAP Core Pack", "plugin": "CoMAP for Skyrim 1.6.x" }, { "step": "Core Files", "grp": "Configs", "plugin": "CoMAP Config Pack" }, { "step": "Undiscovered Marker Options", "grp": "Obscured Undiscovered Options", "plugin": "Obscured Undiscovered Markers - Question Mark" }, { "step": "Addons", "grp": "Khajiit Caravan Markers", "plugin": "Hidden Until Discovered" }, { "step": "Addons", "grp": "Jorrvaskr Map Marker", "plugin": "Jorrvaskr Map Marker (Closed City)" }, { "step": "Addons", "grp": "Other Addons", "plugin": "Ruined Map Markers Use Original Outline" }, { "step": "Addons", "grp": "Other Addons", "plugin": "Altars and Shrines" }, { "step": "Addons", "grp": "Other Addons", "plugin": "Abandoned Mines Use Vanilla Design" }, { "step": "Addons", "grp": "Other Addons", "plugin": "Orc Only Bandit Locations Use Orc Stronghold Marker" }, { "step": "Addons", "grp": "Other Addons", "plugin": "Hearthfire (and Tundra Homestead) Uses Settlement" } ] } } ] }, { "name": "project rainforest se - a tropical climate and environment overhaul", "installers": [ { "h": "CXSrW6W2UMYpPugtFOE33lTMxpEzm2c9VdLiomXKwTg", "type": "SIMPLEUNPACK", "params": { "root": "" } } ], "modtools": [ { "name": "OPTIONAL", "param": { "unopt": [ "project rainforest-dolomite weathers patch.esp", "project rainforest-dolomite-true storms patch.esp" ] } } ] }, ("OPTIONAL" modtool is a trivial thing, moving esps from/to optional folder). NB: currently archives are referenced by their hashes, but replacing them with modid/fileid is not that difficult - we already know all the hashes (including Nexus md5 hashes) for all the archives, so mapping archive hashes to modids should be quite doable.

Thanks a lot for your detailed reply (though on the second thought, it is only logical to expect serious analysis from STEP folks ;-)). Now to answer your questions: yes, it is going to stay open source (in usual sense of this term) " It sounds like a Py program that deductively arrives at cause (mod[s] installation details) from effect (installed mod[s]). I also see mention of post-installation manipulation by applications that modify or augment installed mods (e.g., NEMESIS, BodySlide, ACMOS-RG, DynDOLOD)." - this is what SUMMON code currently is (it was developed as a hobby project, and now we want to make production thing out of it). Side note: from tools - it is currently only BodySlide, and even BodySlide support is still WIP. "But it's also being cast as a mod manager in and of itself.' - and this what we want it to become. TBH, it is not that much additional complexity, compared to complexity of deduction analysis (until we're delving into complexities related to esps - which is a large and rather separate field). And yes, it is going to be in Python too - so it is a natural extension to already-existing code. the advantage over, say, MO2, is two-fold: [most of the time] we know for sure not only what's the archive from which the mod was installed - but also how it was installed (="which buttons were pressed in FOMOD, which esps were made optional using MO2 window, and so on"). This allows to show modders what EXACTLY is going on, and with MO2, it is a problem - in particular, in your regular setup there are usually tons of files within mod folders, which were modified by Bodyslide and FNIS - without you even noticing it, which can have ugly implications and unexplained behaviors; also there are cases of accidental changes made by modders themselves - which can be Seriously Bad if they're unnoticed. To show it, we want to have color codes (or icons) for the mods - indicating "this mod is pristine", or "some of this mod files got modified after install"). more importantly from STEP perspective, these descriptions of "how it was installed" are ACTIONABLE - and are actually "installation instructions". Which means that the whole STEP setup (ok , at first only 80% of it) can be automated. Not sure if you folks like it or not - but as a modder myself, I would LOVE to be able to tell "hey, just download SUMMON, press a button 'I want STEP', and leave it overnight to get STEP on my box, so I can take it as a base for my own modding". We're not there yet - but IMO, it is a Big Thing and Noble Deed(tm). after all, most boring (and making the modder learn the least too) part of the STEP is like "hey, get the latest version of such and such mod, press such and such buttons in FOMOD while installing, remove these 3 files, and add these 2 lines to .INI" - and ALL of it can be done by computer much better than by a human. OTOH, note the difference from Wabbajack here - unlike Wabbajack modlist, SUMMON's "installation instructions" are small, text-based, and most importantly - upgradable (in short - they're just a machine-executable equivalent of the instructions such as in STEP guide). additionally, this opens a door to a fully-github-based modding - where people share these "installation instructions" over GitHub (in SUMMON-speak, these "installation instructions" are named "modpack"), and making another modpack as "use such and such base modpack, adding more instructions on top of it" . This, in turn, will allow to separate responsibilities between different modders, one of them concentrating on environment meshes, others on bodies and clothing, others on lights and ENBs, others on animations, others on quests, and so on - with the modder choosing what they prefer for their setup , as these fields are only loosely coupled (with not that much interaction between them). "Support for creation of automated install guides" - creation of 'guides' (human-readable instructions)? Or perhaps you mean mod 'builds' (VFS configured/installed files/folders)?" - it is automated, but it is seriously different from Wabbajack; see above re. "installation instructions" vs Wabbajack-like modlists. SUMMON's JSON files is like a machine-executable version of human-readable STEP guides. "I don't think we've underestimated." - let's write it down as an agreement about disagreement. And please keep in mind that STEP authors, as the people who are into this for years, are in inherently horrible position to make estimates (you just happen to know way too much compared to Joe Average modder who comes to do it, which means that you're inevitably doing things faster, and without many ugly misunderstandings and silly mistakes, which are quite usual - even though STEP guide is of next-to-impossible quality, kudos for you guys for doing it). "Is the goal to read text guides and convert them into json scripts that are then used by the application to perform the installation(s)?" - that's one way to use SUMMON. And BTW, due to "deduction", most of the time it should be sufficient just to follow the guide to install mod(s) - and then take JSON-generated-by-deduction to make a publicly-available modpack. As I said - I am not sure if you folks will like it, but as a modder I would LOVE to save me some time to do those things which I am interested in doing. After all, let's be honest - STEP is mostly a "Skyrim-as-it-should-be", so why should I spend time fixing Skyrim? NB: SUMMON as such does NOT aim to produce modpacks/installation instructions (!): just is just a TOOL so ANYBODY (hopefully STEP too) would produce them and publish them. We certainly do NOT pretend we have any kind of monopoly on "how to do things" - it is up to seasoned modders to produce such modpacks - and let modders choose. "What about mistakes in instructions and mods? And guide/mod revisions/versioning?" - well, mistakes in modpacks can happen, and they should be fixed via usual GItHub cycle - reporting an issue or make a pull request, and fixing it. As for guide/modpack versioning: new modpack is a new commit to GitHub, so all the usual GitHub things (tags, revisions, etc.) apply; all this stuff was already solved in software development, so we just want to rely on it. As for the mod versioning - by default (i.e. unless specified otherwise in the JSON) we assume that mods are upgradable - as long as the same instructions (such as FOMOD instructions) can be applied to a new version. AFAIU, it is an exact equivalent of STEP guides (which, unlike , Wabbajack, can be applied to upgraded mod versions). "Isn't there still a need for someone to first create/test the construct before SUMMON can be of use in reconstructing it?" - I am not sure whether I understand this question. If it is "is there some magic involved?" - the answer is "no". But if it is "whether a Joe Average can take some modpack (for example, STEP modpack - IF you folks decide to release one, or allow somebody to do it) - and simply run SUMMON to install it overnight automagically" - it should be possible (we're still a LONG way from it, but it is certainly doable). "Is there an AI component?" - nope, there wasn't any need for it (yet?). So far, all the deduction is just a set of heuristic rules, which tend to work very well (ok, it means that, for example, to find FOMOD buttons clicked, we need to run thousands of simulations - but it still works within minutes, and deduction doesn't need to be run too often). "The Git repo is brand new as if it's been developed in a private sandbox up to now, so can we expect to see all development on Git going forward with a listing of contributors?" - it depends , but if there are external contributions - yes, sure, all the external contributions will go via usual GitHub pull requests with contributors being easily visible (it is especially important as we hope for external contributions in particular for plugins, as we won't be able to write all of them, especially the plugins for tools seem to be very involved). Phew, it was long, but it is indeed VERY interesting - what do you folks think about this whole thing?

My apologies if it is prohibited here or a wrong place - please don't hit me too hard if it is so... We're an Irish company of 50+ ppl, working on our own game engine, and as a side - but important - project we're currently developing a new-gen Mod Manager (yes, it IS possible to do significantly better than MO2, and we know how - see our GitHub for details). It is already open-sourced: https://github.com/sixitbb/summon , but before releasing, there is still a lot of things to do. As a result, we're currently HIRING people with the following qualifications: passion for modding extensive modding experience (preferably Skyrim or FO), using mod manager, with at least 100+ mods in a single setup (no, "just this one mod" won't do), and using at least some of the tools such as BodySlide and/or FNIS/Nemesis. Experience with S.T.E.P. is a big plus serious Python experience, preferably production-levell at least understanding why source control is a Good Thing(tm) FAQ: yes, we'll be paying money which are competitive with usual rates for seasoned Python developers - while you will be working on modding (sounds as a dream job, no?  ). position is 100% remote, so self-discipline is Damn Important(tm) on the plus side, you can be anywhere in the world (though being within +-2 hours from Central Europe is a plus) formally - it is a contract with an Irish company with a per-hour rate For further details, for the current code, and for our contacts - please see https://github.com/sixitbb/summon .