Performance Tuning - Crash on Load

[mike10101011]

Alright, so I did a few things here.

1. First I reinstalled Skyrim to the latest version. I only deleted the 02 mods including SKSE. I ran Skyrim on the latest version to download CC.
2. I downgraded using the depot again. I wasn't able to find a 1.6 version of the downgrader, so I did it manually again.
3. I cleaned the masters. There were only 36...not 44, but the guide says "approx" so maybe this isn't an issue.
4. I installed all 02 mods including SKSE.
5. Ran SKSE from MO with only xEdit, Cleaned Masters and 02 enabled and it ran! Great!
6. I actually did have a few issues and wasn't able to simply enable all at once, but with the Extended ESL Support and going through each section one by one, they all ran.

I still need to do the actual Performance Tuning now...but that'll be for tomorrow. Getting a bit late.

[Greg]

10 hours ago, z929669 said:

So "unallocated memory space" probably either means "out of memory" (not enough GPU VRAM or system RAM) OR that "the memory address is just undefined in memory space"? Your guess about Engine Fixes seems to point more to the latter.

Please bear in mind that what I am writing here should approximate correctness as a high level overview, although it is based on my understanding of the implementations and direction Intel was heading up through the Pentium processor in real mode (with an actual memory management unit and flat address space).

It's the latter. Modern processors have an internal memory management unit (MMU) that maps logical memory allocations to physical memory and tracks which areas of memory are allocated to a process and the actions that are allowed in each memory block (execute and read/write). Modern languages like C/C++ and several others also have the concept of a null pointer (0x0) that is used to trap illegal memory accesses (typically because the pointer was never initialized or assigned a valid address). The 64-bit address space is so large (0x0000 0000 0000 0000 - 0xFFFF FFFF FFFF FFFF) and physical memory is so comparatively small (0x0000 0000 0000 0000 - 0x0000 003F FFFF FFFF for 256GB) that this works in our favor.

Note the instruction pointer in the crash is (expanded out to 64 bits) 0x0000 7FF6 DACC C4B3 and (if my math is correct) this is in the 127 terabyte block of logical memory space. I won't go into the details of how the MMU works because I'll probably be wrong at worst and outdated at best. The high level is that when the processor needs to access a logical memory address, it consults the MMU to get the physical memory address in RAM¹. If the logical memory address is not mapped in the MMU, the process raises an access violation exception. This may be naive on my part and is based on observation of a small sample of debugging sessions, but I've never seen a legitimate allocation in the lower logical address space. I believe this is intentional and works to the advantage of address randomization (in which Windows loads executables and DLLs into random addresses to minimize predictable load patterns).

¹This is a very simplified because it does not account for mapping logical address blocks to the page file. The MMU also tracks whether the process is allowed to execute, read or write to this logical address space and throws an exception if you attempt to execute in a non-executable block or write to a non-writable block.

[z929669]

7 minutes ago, Greg said:

Please bear in mind that what I am writing here should approximate correctness as a high level overview, although it is based on my understanding of the implementations and direction Intel was heading up through the Pentium processor in real mode (with an actual memory management unit and flat address space).

It's the latter. Modern processors have an internal memory management unit (MMU) that maps logical memory allocations to physical memory and tracks which areas of memory are allocated to a process and the actions that are allowed in each memory block (execute and read/write). Modern languages like C/C++ and several others also have the concept of a null pointer (0x0) that is used to trap illegal memory accesses (typically because the pointer was never initialized or assigned a valid address). The 64-bit address space is so large (0x0000 0000 0000 0000 - 0xFFFF FFFF FFFF FFFF) and physical memory is so comparatively small (0x0000 0000 0000 0000 - 0x0000 003F FFFF FFFF for 256GB) that this works in our favor.

Note the instruction pointer in the crash is (expanded out to 64 bits) 0x0000 7FF6 DACC C4B3 and (if my math is correct) this is in the 127 terabyte block of logical memory space. I won't go into the details of how the MMU works because I'll probably be wrong at worst and outdated at best. The high level is that when the processor needs to access a logical memory address, it consults the MMU to get the physical memory address in RAM¹. If the logical memory address is not mapped in the MMU, the process raises an access violation exception. This may be naive on my part and is based on observation of a small sample of debugging sessions, but I've never seen a legitimate allocation in the lower logical address space. I believe this is intentional and works to the advantage of address randomization (in which Windows loads executables and DLLs into random addresses to minimize predictable load patterns).

¹This is a very simplified because it does not account for mapping logical address blocks to the page file. The MMU also tracks whether the process is allowed to execute, read or write to this logical address space and throws an exception if you attempt to execute in a non-executable block or write to a non-writable block.

I've taken us off topic here, but thanks for the knowledge dump ... some of it actually makes sense to me

[ButchDiavolo]

9 hours ago, mike10101011 said:

Alright, so I did a few things here.

1. First I reinstalled Skyrim to the latest version. I only deleted the 02 mods including SKSE. I ran Skyrim on the latest version to download CC.
2. I downgraded using the depot again. I wasn't able to find a 1.6 version of the downgrader, so I did it manually again.
3. I cleaned the masters. There were only 36...not 44, but the guide says "approx" so maybe this isn't an issue.
4. I installed all 02 mods including SKSE.
5. Ran SKSE from MO with only xEdit, Cleaned Masters and 02 enabled and it ran! Great!
6. I actually did have a few issues and wasn't able to simply enable all at once, but with the Extended ESL Support and going through each section one by one, they all ran.

I still need to do the actual Performance Tuning now...but that'll be for tomorrow. Getting a bit late.

I see the "Big Hitters" have been responding, so I am sure that the install will work now

[mike10101011]

6 hours ago, ButchDiavolo said:

I see the "Big Hitters" have been responding, so I am sure that the install will work now

They indeed have been helpful. I'm about to start the actual Performance Tuning here in a moment now that the game will actually load!

[ButchDiavolo]

Oh look... Skyrim got updated WHILE STEAM WAS IN OFFLINE MODE!

How is that even possible? I started Steam, then MO2, went in game to test the new addon for Completionist and crashed. The logs showed that somehow I got up****ed updated to 1.6.1170. While Steam has been in offline mode since January 5th or so. *breathes into a paper bag*

Anyway, thanks to this thread I can now downgrade the proper way... again.

[Greg]

As bad as this may sound, it's nice to see confirmation that Steam may mysteriously update itself and games even while allegedly in offline mode. I'm a bit behind at this point, but I currently have backup copies of Skyrim Special Edition 1.5.97, 1.6.318, 1.6.353, and 1.6.640. It looks as if I missed the 1.6.1130 update.

[ButchDiavolo]

12 hours ago, Greg said:

As bad as this may sound, it's nice to see confirmation that Steam may mysteriously update itself and games even while allegedly in offline mode. I'm a bit behind at this point, but I currently have backup copies of Skyrim Special Edition 1.5.97, 1.6.318, 1.6.353, and 1.6.640. It looks as if I missed the 1.6.1130 update.

I always suspected it as well. Having it happen is a bit... disconcerting. I mean, HOW can it update when it is in offline mode. Apparently offline isn't really offline. Thankfully I also had a full back up of 1.6.640.

[mike10101011]

So everything is resolved!

Basically, I kept all of the mods except 02-Extenders. Uninstalled the game. Reinstalled the latest version and launched the game to download all the CC items.

I then used the Download Depot to downgrade to 1.6.640.

Spoiler

download_depot 489830 489831 3660787314279169352

download_depot 489830 489832 2756691988703496654

download_depot 489830 489833 5291801952219815735

Once all that was overwritten, I cleaned the masters. Guide says 44 I think, I only had 39. Pro Tip: backup the Data folder with all the masters BEFORE you clean since you have to move the backups back to Data anyways.

I reinstalled the 02-Extenders including SKSE. I also installed Backported Extended ESL Support  and placed it at the end of 02-Extenders. (I actually forgot and had 7 mods that kept failing until I installed this). I ran LOOT, I had already cleaned other mods from the previous setup. Ran SKSE and it worked!

Moved onto Performance Tuning and everything else fell into place.

@DoubleYou@z929669@ButchDiavolo Special shout out to y'all for helping out here. Thank you!

[z929669]

39 is correct for 1.6.1170.

We do note all possibilities, albeit, it's arguably TMI and convoluted.

SE guide:

SSG ("Verify the integrity of game files" instructions need minor update to Steam's updated UI):

Back to the SE guide: It's easier to explain and support using Steam's validation to restore the files rather than instructing people one of the several ways of backing up files via the OS. Besides, it's always prudent to verify the integrity of the game cache via Steam when you're messing with the game files.

 

UPDATE: I have my Steam Updates for SSE set to "Always keep this game updated". I had auto-updated to 1170 and cleaned all masters in the last couple of days, and all was kosher. At that time, LOOT flagged 39 masters as having ITM/UDR, so I thought I had cleaned them all, copied anew into my MO mod, and used Steam's "Verify the integrity of game files".

Tonight, I was surprised to find that I only had 37 "cleaned vanilla masters" in my MO mod and my game Data directory. From Steam Lib while typing this post, I noticed that these cleaned mods were not considered in "Verify the integrity of game files", as they were still identical to my custom cleaned versions in MO. Further, I witnessed that "Verify the integrity of game files" was not reacquiring the original masters, I deleted the now 37 masters that I had cleaned from Data, but they still weren't reacquired. I had to actually launch the game via Steam, and click "Play" to invoke download of ALL apparently 74 CC masters.

THEN, I had another look at the game Data directory and verified the timestamps of the now 74 CC masters. LOOT tells me that only 28 of them have ITM/UDR, so either LOOT data has changed, or the vanilla masters have changed in the last 48 hours. Spot checking the Created dates of some of those original 39 I had and had not updated in Data showed that the game's "Download" process reacquired those I had deleted and only 'touched' those I had not (all 74 modified dates were 'Today', but only the 37 reacquired ones had Created dates in the past).

... so I think we either need to simplify our instructions or add some additional ones explaining the additional steps and possible caveats.

 

[Mousetick]

AE CC modules are downloaded from Bethesda.net servers by SkyrimSE.exe, not from Steam servers by the Steam client. Steam doesn't know about these files, so they are ignored by 'Verify integrity of game files' and it has no effect on them.

[z929669]

3 hours ago, Mousetick said:

AE CC modules are downloaded from Bethesda.net servers by SkyrimSE.exe, not from Steam servers by the Steam client. Steam doesn't know about these files, so they are ignored by 'Verify integrity of game files' and it has no effect on them.

And in the far recesses of my mind I knew this, but our instructions on need to better reflect it. It's easy to forget the details when it's only done once every year or two

[VosslerWeiss]

On 2/2/2024 at 6:18 PM, DoubleYou said:

Install this: https://www.nexusmods.com/skyrimspecialedition/mods/106441

Thank you for this. This really helped me after Steam forced update my 1.6.640 to the latest version then I had to downgrade it back to 1.6.640 but that broke my mods, especially sse engine fixes which was the main culprit. Installing this solved the issue for me for now.