Automation Survey

From Step Mods | Change The Game


Vista-file-manager.png Archived Page
This page has been archived. The contents have been moved from another page for reference purposes only, and should be preserved in their current form. Any additions you make will probably not be read and may be undone without notice.

Overview

At the time of writing, the process of setting up STEP can be quite painful. There are many manual steps in the process. Some of these steps have to be done in a particular order, but others can be done in parallel. Some of these steps have to be done manually, whereas others can be automated. The goal of this page is to do a survey of what it takes to go from "I just bought Skyrim Legendary Edition on Steam!" to a working STEP system (with zero or more packs installed on top of STEP Core, perhaps with different Mod Organizer profiles), and what parts of that process can be automated.

Glossary

Special terms used on this page:

  • [RR] = Research Required. This section can be improved by spending some time looking into specifics.
  • [N2H] = Nice-to-Have. This idea could be skipped entirely, and most users probably wouldn't mind.
  • [INC] = Incomplete. This section is just a placeholder and needs to be filled in.

Automation

In general, "automation" here refers to using a tool of some sort to perform a process without user interaction that would have otherwise required it. Automation is highly desirable for STEP for these reasons:

  1. The automated parts should run much faster than otherwise
  2. Reduces the number of ways users can make mistakes during the setup process
  3. Reduces the burden on a user to have to learn how to use several different power-tools
  4. Simplifies the setup process, bringing in more users; some of these users may become contributors themselves

Automation does, however, have some disadvantages:

  1. Expensive to set up in the first place: the time we spend automating STEP is time we could have spent improving something else
  2. May not help "advanced" / "power" users who want to deviate slightly from what the guide recommends

In general, on a technical level, the entire process can be fully automated. Any tools that need to be run through a GUI can be automated via AutoIt or AutoHotKey scripts, if all else fails. The only barriers to automating the full process of setting up STEP are non-technical (lack of resources / social).

STEP Setup Process

At a high level, this is what the STEP setup process looks like. High-level sections should more-or-less correspond to the STEP guide.

Install Skyrim

Install the latest version of Skyrim, with all official DLC, cleanly (no local modifications).

Automation Opportunities?

Assuming that the user has purchased a legitimate copy of Skyrim with all official DLC, Steam handles this entire process for us. Still, this step requires transferring multiple gigabytes of data over a potentially slow network connection and writing it to potentially slow persistent storage, so there are some opportunities here.

  1. [RR] Can SteamCMD be used to check if Skyrim is owned with all DLC, download if not present, and verify local cache if it is present? If not, we can fall back to AutoIt / AutoHotKey scripts to run through the Steam client UI, but that feels tricky and fragile.
  2. [RR] / [N2H] Can we install a separate instance of Skyrim to cover cases where users have been having a jolly old time with their install, maybe even with a few Steam Workshop mods here and there, and don't want to abandon their old saves etc. until they've had some time playing around with STEP? Piggy-backs off of the SteamCMD idea above.

Initialize Basic Structures (Skyrim Launcher)

Initialize the structures that the Skyrim Launcher initializes when it starts up: registry keys, baseline INI files stuff.

Automation Opportunities?

We can automate the creation of the corresponding registry keys, but that feels sketchy. Having a one-off AutoIt / AutoHotKey script open the launcher and click through to exiting it probably wouldn't be the end of the world.

INI file stuff might be handled by the section that's dedicated to INI tweaks, but we might as well copy the baseline INI files to the place where the Skyrim Launcher would have copied them, because it's really awkward trying to launch vanilla Skyrim using TESV.exe without ever having run the launcher.

Install ENB

Grab ENB and extract it into the Skyrim install directory.

Automation Opportunities?

We may be able to distribute / redistribute ENBSeries, but the user must have the ability to choose either "wrapper" or "injector". Different software (and hardware?) configurations will likely push different users more towards one solution than towards the other.

We probably also want to make it easy to switch from "wrapper" to "injector" and/or vice-versa.

Tweak INI Files

Change vanilla configuration options stored in INI files, as well as enblocal.ini for ENBoost.

Automation Opportunities?

Easy to automate without having to open the launcher. We can probably just distribute fully baked INI files crafted for whatever setup we're automating. If that's not cool for one reason or another, we can probably have the tool start from the provided baselines instead and apply specific changes to the defaults instead.

Update & Tweak Graphics Drivers

Get the latest-and-greatest graphics drivers for your system, and tweak certain options according to the STEP guide's recommendations.

Automation Opportunities?

We're pretty limited here, and all the work has to be done pretty much twice (once for AMD, once for NVIDIA). At best, we can maybe detect AMD / NVIDIA and point the user to the corresponding wiki page. Fortunately, this can be done pretty much any time before the user goes to run the game for the first time, so this step isn't a blocker for the rest of the setup process.

  1. [RR] / [N2H] Maybe a pack can say "when this was released, the latest AMD driver version was FOO and the latest NVIDIA driver version was BAR", and then if we detect a lower version, we warn the user?
  2. [RR] / [N2H] Could we maybe host a web service that just provides a static JSON document describing the latest graphics driver versions for AMD / NVIDIA?

Calibrate Display

Make it so your computer screen shows the same lights as other people's lights for a given picture shown by your computer.

Automation Opportunities?

See comments on Graphics Drivers. This can even be skipped entirely without much harm.

Download Mods & Third-Party Tools

Grab the latest or specified versions of over 100 files, nearly all of which are hosted on Skyrim Nexus.

Automation Opportunities?

As painful as this process is for users, and as (comparatively) easy as it would be for us to automate, opportunities are limited on this one. According to MontyMM in forum thread "Skyrim Nexus Downloader" (direct link), Dark0ne (the owner and administrator of all Nexus sites) has explicitly discussed his wishes for automated downloading: 

   I discussed automated downloading with Dark0ne, and it's not something he would like us to do for the foreseeable future. I feared as much, and it would indeed screw up the way the Nexus registers downloads, endorsements, etc. He suggested we could do something similar with a plugin for NMM, but that this might mean reduced ad revenue, with fewer eyes on the site. I told him we wouldn't want to do anything that harms the Nexus, so I think automatic downloading should be considered a non-starter. I did, however, suggest that he might like to implement mod collections formally on the Nexus, with perhaps automatic downloading of the collections as a premium feature. I'll let you know what he thinks of that.

Unless / until Dark0ne pivots on that, STEP solutions should respect that. Skyrim Nexus has been a major part of the STEP story, and it would be extremely impolite to go against the owner's wishes.

What we can do here without being bad people:

  1. Detect which archive files we don't have to download, because we've already downloaded the same version earlier and we haven't deleted it yet.
  2. Point the user's web browser at one or more mod pages on Skyrim Nexus, with instructions saying which exact files to download.
  3. Download from sources who are explicitly open to automation, where the mod author has given permission (many mod authors forbid rehosting their mods outside of a particular source).

Updated 2019-04-12: Dark0ne has pivoted in this Reddit comment (archive link, in case it gets edited). Linking to direct downloads is now OK as long as you use the API (which, at the moment, restricts it to Premium users only).

Unpack / Install Mods & Third-Party Tools

Extract subsets of the contents of archives downloaded before, to various locations. Typically, this is a location within Mod Organizer's "mods" folder, but a few things are special.

Many mods include a "fomod", which effectively allows a user to interactively choose what subsets of the archive to extract to which target locations in a very friendly way.

Automation Opportunities?

This is likely the most painful part of the process that automation can improve significantly.

Things to touch on ([INC]):

  1. Handle "simple" archives that Mod Organizer would be able to automatically install without even asking the user to set the Data directory.
  2. Handle archives that contain a single "Data" directory and perhaps some loose README-like metadata files.
  3. Handle complicated archives with different sets of options, likely with a "fomod". Most likely, the pack creator would need to be able to describe exactly which files go exactly where.
  4. Handle mods with multiple archives that expect to be installed via a "Merge".
  5. Consider cases where mods do not need to be modified at all between STEP version upgrades, and thus the user does not need to do anything about them.
  6. Consider mods that get installed in "Configuration A" for STEP Core, or "Configuration B" for a STEP Pack.

Set Load Order

Run LOOT with special overrides, and accept the load order it generates.

Automation Opportunities?

It actually wouldn't be the end of the world to just have a pack say "here's the load order before running patches", "here's your load order after running patches", and call it a day. Mod Organizer just looks like it uses "loadorder.txt" and "plugins.txt" files.

[RR] Barring that, it looks like LOOT is a web app with its own little browser framework going on, so maybe we could use some browser automation hooks... or just AutoIt / AutoHotKey.

Apply Patches

Run a sequence of external tools to generate extra data on-the-fly given the specific set of mods installed and their load order. This is:

  1. Run SkyProc Patchers
  2. Build a bashed patch
  3. Generate DynDOLOD stuff

Automation Opportunities?

For SkyProc patchers, there's a lot here that we can do. We can run binaries through Mod Organizer from the command line. We can use an AutoIt / AutoHotKey script to have it run the patcher once its GUI opens (or maybe there's a way to tell the patcher not to open a GUI and just do the thing?). We can even try to speed it up by running them one-by-one through SUM.

[RR] For the bashed patch, there doesn't seem to be much opportunity to build this easily automatically. AutoIt / AutoHotKey scripts feel like they would get pretty complicated. It happens later than most steps, so it wouldn't be a deal-breaker if this had to be a manual "stop the world until the user clicks these buttons" step. An alternative if we can't get absolutely anything working might be to just distribute / host pre-cooked bashed patches and say "if you change anything, rebuild it yourself".

DynDOLOD 2+ looks easy enough to automate, even if it may have to be AutoIt / AutoHotKey.

Add Packs (if any)

[INC] Write something about ENB presets here.

[INC]

Automated (Partial) Solutions

What solutions do we have to some of the problems listed above?

StepperUpper (by airbreather)

Automates verifying downloads, downloading the few files that can be downloaded automatically, extracting archives, setting up Mod Organizer profiles (mod list order, load order, executables), and cleaning dirty files (including Bethesda ESMs). Also semi-automates creating a Bashed Patch, by allowing the pack creator to embed it right in the file.

Requires hand-editing XML files that define the pack, which is very tedious.

Status

Released, no longer being actively maintained as of October 2017.

Capable of setting up STEP 2.10.0, both Core and Extended (as a pack), minus these steps of the process:

  1. Downloading files from sites that don't want us hotlinking
  2. Installing Steam + Skyrim + its DLCs in the first place
  3. Setting up the registry files and other one-time stuff that the Skyrim Launcher does on the first run
  4. Driver settings from 1.E of the STEP guide
  5. Getting the best numbers into the [MEMORY] section of enblocal.ini
  6. Creating custom LOOT meta rules
  7. DynDOLOD output creation (which has other subtle consequences)
  8. Dual Sheath Redux patch creation
  9. FNIS behavior file creation

YAMM (by Terrasque)

This looks like a semi-automated way to solve the concerns of downloading a number of mods (section "Download Mods & Third-Party Tools", above).

Status

Beta.

[RR] Without using it myself, it looks like it addresses the core concerns of downloading mods and their dependencies, but it may be a bit fragile.

The GitHub project project was last updated November 2015. Terrasque has since noted: 

   I'm afraid there's too much politics and "this is MY land!" thinking going on to make anything like that a reality and still go over well with the community.
   
   I'm still working on my system a bit now and then, but I don't think it will see any real use.

SASTEP (by MilesTeg)

This looks like a hodgepodge of different tools designed to solve different highly technical parts of the puzzle, some of which are only useful for pack developers.

Notably, the STEP wiki page seems to have had at least some goals that overlap with the goals of this page, but it looks abandoned after little more than a shallow dive into the realm of what's actually needed by the users.

  1. getMeta translates data from Mod Organizer form into a .csv file. Marginally useful for pack developers, not useful for users at all.
  2. datatreecmp looks at an archive file and a folder where some of the data from the archive was extracted, and determines how it could have been done. Perhaps very useful for pack developers, not useful for users at all.
  3. generateCoreBookmarks seems to create bookmarks from download URLs. Not particularly useful for pack developers, marginally useful for users. Likely obsoleted by YAMM above.
  4. readBCF just translates a .bcf into a user-readable format. Perhaps very useful for pack developers, not useful for users at all.

Status

MilesTeg has not updated this project since late December 2013, and at the time of writing, it appears that this user has not been on the STEP forums since January 2014. Nobody appears to have touched the project since then. There does not appear to be much in a "complete" state that's relevant to the project today.

ASI: Automated STEP Installer (by mothergoose729)

Despite its broad name, this just took care of the "Unpack / Install Mods & Third-Party Tools" section. It also doesn't support the current standard of using Mod Organizer: 

   I plan on adding MO integration to ASI as soon as I have time. I think it might be possible to add mods into MO exactly as MO does it using the meta files. This would allow the users to have the same level of control and features they would have if they did the installation through MO themselves. ASI, in its current form, however, cannot be updated and individual components cannot be removed, that is true.

Status

mothergoose729 has not posted in the project thread since late November 2013, and at the time of writing, it appears that this user has not been on the STEP forums since January 2014. Nobody appears to have touched the project since then. Given the lack of Mod Organizer support and the absolute necessity of Mod Organizer when it comes to STEP packs, this is almost entirely unfit for current STEP use.

Mod Combiner (by generalmx)

This appears to just solve the same problem as ASI, the same basic way (so no Mod Organizer support).

Status

No apparent updates since February 2013.

STEP Thunderbolt (by frihyland)

This one doesn't really belong on this list because I can't find concrete evidence of even an early version of even part of this tool.

According to legend, this was intended to be a one-button STEP setup application.

Status

Nothing has ever been released, and there's no evidence of progress being made on this.

Retrieved from "https://stepmodifications.org/wiki/index.php?title=Automation_Survey&oldid=147028"