CTD at 3.1GB

[viking]

Some key posts on this and related threads (experts feel free to note any errors or insights):

• Post #11
• Post #27
• Post #40 (this is where the 500/430 ratio seems to have originated.)
• Post #66
• Post#278
• Post #313
• Post #1 (adjacent thread: Caution about memory optimisers)
• Post #64 (adjacent thread: Caution about memory optimisers)
• Post #68 (adjacent thread: Caution about memory optimisers)
• Post #70 (adjacent thread: Caution about memory optimisers)

Wiki article (draft)

 

Thanks,

STEP

 

OP follows

---

First of all I wanted to thank you all for the great work you have done with STEP. Skyrim is the first game I installed on this computer and you guys have made it an AMAZING game. That being said, I have an issue that I hope you can help me solve.

 

My setup:

Vanilla Skyrim

gtx670 w/ 4GB @1080p w/ latest driver

16GB Memory

3770K at about 4GHz

Windows 8 64bit

ENB 149

Ultra settings

Highest available texture/quality

Mod Organizer

Step 2.2.1 + Skyrim Revisited + others

 

I have noticed a post here and there saying that Skyrim can't really address more than about 3.1GB of memory without issues. This seems to jive with my experience, meaning I CTD every time my memory hits that mark, but I couldn't really find anything definitive on the topic. The issue with googling the topic is the pre 1.3 skyrim that couldn't address more than 2gb of memory.

 

My mod list is mostly based on STEP which is why I came here for help, with about ten mods added onto the end (Interesting NPCs, Detailed Cities, Economics, COT, and a couple others). The reason I haven't included my mod list is that it doesn't seem to matter. As long as I keep the memory usage below 3GB I can have pretty much any combination of mods.

 

What I have tried so far (in no particular order):

• resetting ini files
• removing enb
• not using attklt
• only using a new game
• removing all mods and adding one by one until issue crops up
• running as admin
• watching the papyrus log - it seems relatively clean, no obvious errors right before CTD

Yes, I can run STEP just fine without any issues, but I also never get near 3GB of memory. I have tracked VRAM usage as well and have seen a max of 2.7GB/4GB.

 

As an example of where I might run into issues: I start a new character with Alternate Start. I start with Breezehome. Run out of Whiterun, past the Brewry, up the hill to the bandits. Enter the cave (watching memory usage with Elys MemInfo), and it dies right after I see 3GB. I have this same issue not using AS, sitting through the intro, and then running over to whiterun.

 

I'm sorry if this post is all over the place. I have spent more than a week trying to solve this issue, and the only solution I have found is to reduce memory usage. I have got to the point where I can exchange two texture packs and get into the cave without a CTD, but with both I get a CTD. I didn't even think texture packs should even affect CTDs, but I'm relatively new to Skyrim on the PC, so I could be wrong. I also found I could get a bit further with ENB turned off, but would still crash once I got above 3GB of memory. Finally, if I reload a game after a CTD, I can play just fine...until I reach 3GB of memory.

 

I really hope you guys can help. I more than willing to try anything at this point, besides just disabling all of the mods.

[Aiyen]

Should not matter no. The texture resolution etc. remain the same. More stuff just needs to be rendered at any one point so the GPU/CPU needs to do more computing, but that is about it.

[z929669]

Frame rate definitely takes a hit, and AFAIK so does VRAM (although I cannot remember testing this explicitly, I feel like I have noticed this).

 

Check Vond's post though, which seems to corroborate

[Aiyen]

Yeah VRAM would increase. Especially if you use ENB since the size of the AO filter is based on screen resolution, so a larger AO texture requires more VRAM since it is stored there.

 

I guess there are other "maps" that also have to be stored in VRAM that will also increase with resolution.

But as if those are also mirrored in RAM I do not know for sure. But at least the ENB ones do not seem to be... at least I do not see a 100Mb drop in RAM use when I enable or disable ENB.

[GrandBulwark]

I found that even with an impeccable system; the memory will become an issue if you envelope the world with 2048 textures. That being said, even cutting half of said texture to 1024 will greatly improve framerates and stability. My suggestion would be to find every predominant outdoor texture pack (eg. landscape,flora and water) that you are currently using and see if they offer high performance versions. That being a better option than optimizing said textures as in certain cases that can greatly impact graphic fidelity. There are also a few textures that do not like being optimized at all, that is usually in relation to weapons and armor in my experience though.

 

I found this out the hard way. Granted through the use of my guide; I wouldn't crash. However, the framerates drops were irritating. All I did was switch from SRO 2048 to 1024, luckily I have both backed up on my backup HDD. Also isolating recurring instances of 4096 textures can greatly help. Once in a blue moon is fine, but if your fur armor is a 4096 for every piece; your computer is the only thing that suffers when bandits attack you. ;)

[timaishu]

I just installed a new 7950 3gb card and therefore, decided to redo my step install to install higher resolution mods. I decided to take the highest available for everything. I began noticing random CTDs. Came across this thread I believe this must be the issue.

 

I am having trouble measure vram usage on the fly because it doesn't always work with skyrim performance monitor(assuming because of my enb). Last CTD, I had gpuz running and my max usage for that session was 2.8gb of vram.

 

Gonna give this thread a read, but I hope I don't have to get rid of my pretty textures. :(

[phazer11]

NEVER EVER take the highest resolution of anything. At most you might (big might) be able to have one or two 4k textures and that'd likely be pushing it. Try the 1k textures of everything then if everything is stable for a while try adding a few 2k textures (a few) and playing with that for a while. Skyrim is a 32-bit game and mirrors part of your VRAM usage into RAM (this is on top of whatever the mods with scripts you are using and the game engine eat up).

[Aiyen]

Deleted double post

 

Sadly you cannot really get away with even 2k textures in the exteriors. It might look pretty, but at the end of the day if you want to do more then take pretty screenshots then you might as well downscale.

 

Sadly texture makers do not seem to remember/care that people are going to use their work in combination with other similar texture files. Which all together is going to make it so the game cannot even run.

Also after a while you are going to stop noticing the lower resolution textures regardless. If you want to mimic the feel of them then I can suggest that you just try to increase the contrast setting of your ENB ever so slightly. This will make the textures stand out more as well.

Alternatively then also use the SweetFX texture sharpening addition which will also help make lower res textures look a bit better.

[Wormheart]

If only we could batch resize the textures based on the sizes used in the HRDLC, some sort of frontend to ddsopt that uses a lookup table or something. It would certainly help cut down those mods that have new textures that are 4 to 16! times the size of the vanilla texture.

[grhuff]

EDIT #1: Just to point out the obvious, but please make the appropriate backups of your Skyrim folders before running these tools just in case you get an unexpected result. I think you should be okay, but there is a good chance your install is not exactly the same as mine and you may be using mods and textures that I'm not. So please make backups just in case.

 

EDIT #2: If you're not using Mod Organizer, then your procedure for #5 will be a little different and you will need to review the .ini file to make the necessary changes to reflect whatever your folder structure is. Some of the references to specific file types and folders may not be the same if you're not using MO.

 

I was waiting to make a post with my results on this topic, but I'm now able to run all of the 2048 texture packs (Skyrim HD, Skyrim Realistic Overhaul, and Serious HD) without CTDs or freezes. Keep in mind that I use a base install of Skyrim Revisited with a few STEP-specific and other mods sprinkled in. This guide is assuming you already have a base install using STEP or SR and are using Mod Organizer to manage it. I can provide a more thorough guide for this in a separate post, but here is the general recipe for my success:

• To run all of the 2048 textures you will need a GPU with at least 3k VRAM - if not, I would not try to run the 2048 texture packs but you might still get mileage out of the rest of this guide
• Extract and optimize the Bethesda textures with DDSOPT per Neo's guide
• Add the HRDLC Optimized (Hybrid + Vanilla normal maps option) just before the big texture packs in your load order
• Run your entire Skyrim folder through SMCO
  • It has 32-bit and 64-bit versions, so use the one that matches your system
  • *I only recommend using the DDS tab, not NIF or BSA
  • Point it to your install folder (something like C:\Program Files (x86)\Steam\SteamApps\common\Skyrim\)
  • Click the blue "update base exclusions" button (I do this everytime I run the tool just to be sure)
  • Choose the number of threads (default is 4, but I do 8 - depends on your system resources)
  • Click Optimize
  • When it's finished, I usually use the "Logs and Hashes" button to clear the HTML logs and hashes so that SMCO runs a full scan every time
  • In addition to doing this whenever you modify mods with textures, you may want to repeat this entire step a second time because I've noticed some textures can be further optimized by subsequent passes
• Run your Mod Organizer/mods folder through Optimizer Textures
  • You need to use a modified Ordenador.ini for best results - I've tweaked mine to account for the use of Mod Organizer and also some SR/STEP-specific file exclusions (listed below)
  • You should run this as Admin
  • Point it to your MO mods folder (something like C:\Program Files (x86)\Steam\SteamApps\common\Skyrim\ModOrganizer\mods\)
  • Under Profiles, select "Maximum", then select "Skyrim" in the second dropdown
  • *I only recommend using the Textures tab, not BSA
  • Click Start
  • This can take a while, but when it's finished I also recommend pointing it at your Automatic Variants texture override folder, assuming you're using AV (C:\Program Files (x86)\Steam\SteamApps\common\Skyrim\ModOrganizer\overwrite\textures\)

This is my Ordenador.ini file. I modified it to account for the use of Mod Organizer along with several other tweaks for files installed during SR and STEP. I recommend reviewing the "Last_Folder", "Backup", and "Multitask" lines at the top to customize them for your install and system. The Multitask line is the same as the threads in SMCO. As you're running this tool, if you see errors in the log about unknown file types, you can add them as exclusions in the .ini file using the template "Preserve_File = *.ext". I'm pretty careful about what I install in MO, so my .ini may not cover everything. The error messages are harmless but I prefer to prevent them if possible. As you use this tool, if you notice any textures that have been incorrectly optimized, you can restore them from a backup and make any necessary edits to the .ini to prevent this in the future. However, I've found this .ini to have the results I expected.

 

Last_Folder = C:\Program Files (x86)\Steam\SteamApps\common\Skyrim\ModOrganizer\mods\
Last_Folder = C:\Program Files (x86)\Steam\SteamApps\common\Skyrim\ModOrganizer\overwrite\textures\
Last_Folder = C:\Program Files\Steam\steamapps\common\Skyrim
Last_Folder = C:\Program Files\Steam\steamapps\common\Fallout New Vegas
Backup = 7z = D:\Data\Downloads\Games\RPGs\Skyrim\Backup\Textures Backup\
Keep_DateTime = True
Multitask = 8
================================================================================
BSA_Compress = NIF KF HKX TRI DDS BMP TGA PNG STRINGS XML PEX PSC CFG TXT DOC
BSA_NoCompress = WMA WAV FUZ xwm OGG MP3 MP4 MPG MPEG AVI FLV WMV BIK JPG JPEG ZIP RAR Z 7Z
================================================================================
[COMMON]
Preserve_File = * - Animations.bsa
Preserve_File = * - Interface.bsa
Preserve_File = * - Meshes.bsa
Preserve_File = * - Misc.bsa
Preserve_File = * - Shaders.bsa
Preserve_File = * - Sounds.bsa
Preserve_File = * - Textures.bsa
Preserve_File = * - Voices.bsa
Preserve_File = * - VoicesExtra.bsa
Preserve_File = *.png
Preserve_File = *.ini
Preserve_File = *.esm
Preserve_File = *.esp
Preserve_File = *.json
Preserve_File = *.txt
Preserve_File = *.jpg
Preserve_File = *.jar
Preserve_File = *.htm
Preserve_File = *.reroute
Preserve_File = *skyrimcloud*
Preserve_File = *\FaceGenData\*
Preserve_Folder = *\textures\MGE\
Preserve_Folder = *\Textures\sky\
Preserve_Folder = *\Textures\Water\
Preserve_Folder = *\textures\effects\
Preserve_Folder = *\interface\
Preserve_Folder = *\scripts\
Preserve_Folder = *\SKSE\
Preserve_Folder = *\meshes\
Preserve_Folder = *\seq\
Preserve_Folder = *\sound\
Preserve_Folder = *\strings\
Preserve_Folder = *\Readmes\
Preserve_Folder = *\optional\
Preserve_Folder = *\Docs\
Preserve_Folder = *\tools\
Preserve_Folder = *\music\
Preserve_Folder = *\Video\
(min, med) Set_ARGB = *\textures\actors\character\*\*_msn.dds
(max) Set_R5G6B5 = *\textures\actors\character\*\*_msn.dds
(min) Set_ARGB = *_msn.dds
(med) Set_R5G6B5 = *_msn.dds
Set_NoResize = *\interface*
Set_NoMipmaps = *\interface*
Set_NoResize = *\icon*
Set_NoResize = *\menu*
Set_NoMipmaps = *\menu*
Set_R5G6B5 = *_n.*
Set_NoResize = *\textures\clothes\cloaksofskyrim\Cape_n.dds
Set_NoResize = *\textures\clothes\cloaksofskyrim\sagecloak_n.dds
(min) Preserve_Folder = *\Textures\apachii*\
(med) Set_ARGB = *\Textures\apachii*\
(med) Set_NoResize = *\Textures\apachii*\

[NEW VEGAS]

[SKYRIM]

[MORROWIND]
Preserve_File = textures\_land_default.tga
Set_Icons32x32 = Icons\*
Set_NoResize = BookArt\*

[OBLIVION]
(min, med) Preserve_Folder = Textures\Menus\
(min, med) Preserve_Folder = Textures\Menus50\
(min, med) Preserve_Folder = Textures\Menus80\
Set_Icon64x64 = Textures\apachii\menus\icons\*

[STALKER]
Set_R5G6B5 = textures\clutter\common\hotiron01.dds
Set_R5G6B5 = textures\black.dds
Set_R5G6B5 = textures\brightyellow.dds
Set_R5G6B5 = textures\default_n.dds
Set_R5G6B5 = textures\defaultdiffuse.dds
Set_R5G6B5 = textures\defaultwithspec_n.dds
Set_R5G6B5 = textures\gray.dds
Set_R5G6B5 = textures\green.dds
Set_R5G6B5 = textures\greynoise.dds
Set_R5G6B5 = textures\lightblue.dds
Set_R5G6B5 = textures\white.dds
Set_R5G6B5 = textures\white_em.dds

[FALLOUT 3]
Preserve_File = Textures\gore\fxbasegore.dds
Preserve_File = Textures\gore\gorearmparts.dds
Preserve_File = Textures\gore\goreheadbits.dds
Preserve_File = Textures\gore\goreintestine.dds
Preserve_File = Textures\gore\gorelegparts.dds
Preserve_File = Textures\gore\goretorsobits01.dds
Preserve_File = Textures\gore\screenbloodcolor01.dds
Preserve_File = Textures\gore\screenbloodcolor01opt.dds

[Aiyen]

Nice addition with some good points. However I got a few issues.

 

Why would you run both DDSopt and Optimizer textures? DDSopt can do all the same operations that OT can and then some.

 

I have not yet done anything with SMCO, so I will give it the benefit of a doubt. But again anything that "optimizes" textures essentially do the same operations as DDSopt and should as such not be necessary.

 

As for the mesh "optimizations"... then I will have to do more research before commenting. But my initial impression is summed up nicely in the known issues of the mod. If you perform this on your entire MO folder and you get CTD´s due to meshes messed up, then you will have a really hard time finding out the correct cause.

[grhuff]

Nice addition with some good points. However I got a few issues.

 

Why would you run both DDSopt and Optimizer textures? DDSopt can do all the same operations that OT can and then some.

 

I have not yet done anything with SMCO, so I will give it the benefit of a doubt. But again anything that "optimizes" textures essentially do the same operations as DDSopt and should as such not be necessary.

 

As for the mesh "optimizations"... then I will have to do more research before commenting. But my initial impression is summed up nicely in the known issues of the mod. If you perform this on your entire MO folder and you get CTD´s due to meshes messed up, then you will have a really hard time finding out the correct cause.

You are correct, trying to optimize meshes (or NIFs as I mentioned in step #2 under SMCO) is probably not a good idea and one that I would not recommend. The author of SMCO even states this on the mod page. The only reason I ended up using DDSOPT at all was because I followed Neo's guide for SR where he provided very good instructions. It seems easy to make mistakes using DDSOPT. I haven't noticed any issues with then running the entire texture install through SMCO and Optimizer Textures. As far as using DDSOPT alone, that would probably work provided you took the time to set it up properly and were selective in what you targeted. In my opinion, the benefit to using tools like SMCO and Optimizer Textures is that they are simpler.

[axle356258]

Hello everyone. I have been reading the articles on the Skyrim 3.1 gb limit. 

 

I have a decent system running:

 

intel i5 3570k quad @ 3.4ghz

2 x Geforce gtx 670 @ 4gb ea.

3 x Asus 27" Monitors

32 gb ddr3 sdram

Windows 7 pro. 64bit

Asus eK Quad X Motherboard

Etc. Etc. Etc.

 

I'm running a heavily modded Skyrim with A LOT of Texture mods. most of them between 2k and 4k graphics. 

Haven't been one to use the S.T.E.P. program yet. (Looking to soon)

 

Ok now to the point, with the system I have, one would expect to not have any ctds due to performance but, I WAS. It was very frustrating.

Then one day a friend of mine made a simple suggestion to me that was almost too obvious. 

Move the Skyrim client folder out of the 32 bit Program File (C:\Program Files (x86) folder just onto the operating system root drive ( ie. C:\Steam\Steamapps\common\Skyrim\TESV.exe)

(of course, because I have steam and play through the steam app, that meant un-installing the 50+ games and moving the whole thing.. took a few hours)

But once done, I haven't had anymore performance crashes since. and my monitors have told me at places I've hit just over 4.2 gb at times. 

 

Also another thing to look at is that Windows (any version) has always only allowed a certain percentage of usable memory to be allocated to any one running executable at one time. 

I'm not sure what the exact number is but it's roughly around 24% to 26% of your physical RAM. so if you are running 4gb ram then Skyrim will have issues around 1gb, 8gb around 1.5 - 2.2 gb, 

16gb around 2.9 - 3.8 gb, 32gb around 7.4 - 9 gb.

having more RAM does raise the limit. but that also goes for any Program you run that may be cpu intensive ( depending also on your GPU).

 

Running any program outside the C:\Program Files ((x86)) may also help a lot in the performance because you don't have Windows telling it how much RAM or VM it's allowed to have, to an extent. 

 

I know this because I still have a working IBM 286 with Windows 3.1 on it and it does the exact same thing with the programs that were written for that era of processing power.

[rolandito]

Umm, no.  Please do not post authoritative-sounding treatises on computer memory architecture without knowing the material yourself.

 

Windows has never limited single programs to a fixed percentage of physical RAM.  In the era of 32-bit architectures, Windows NT and its progeny used a virtual memory architecture that presented every process executed with 4GB of addressable memory space.  In other words, each program believed it was running by itself on a machine with 4GB of RAM installed.  How the virtual addresses mapped to the physical memory was handled by the Windows memory manager, and it often involved using hard-disk space as temporary storage (the paging file).  As the concurrent running processes' memory demands approached the limit of installed RAM, Windows would attempt to swap more and more memory data to the paging file, causing thrashing and eventually out-of-memory errors that resulted in process crashes or system lockups.

 

The 4GB "limit" existed because that's the maximum range that could be described (or addressed) using 32 bits when each unique combination of bits references a unique byte of memory.  [2 raised to the 32nd power is 4,294,967,296]

 

In the mid-'90s, the Physical Address Extension (PAE) architecture enhancement was developed that enabled CPUs to use 36+ bits for addressing physical RAM, up to 64GB...however, the Windows virtual memory addressing scheme remained at 32-bits, meaning each process could "see" a maximum of 4GB.

 

In reality, each system required dedication of some memory for video addressing, memory-mapped I/O, and other system tasks, effectively limiting 32-bit personal computers to 3.0 - 3.5GB of usable physical RAM.  Microsoft also introduced an artificial limit in their non-server 32-bit Windows operating system code that locks out addressing above 4GB.  (See "3GB Barrier" for more detail.)

 

In 64-bit land, none of this matters.  CPUs enter "long mode" and can address a theoretical maximum 16-exabytes of RAM when using all 64 bits, and the virtual process space in 64-bit Windows uses 48 bits for addressing, allowing each process to "see" 256 terabytes of available memory, while continuing to use mapping tables and paging file where needed to translate to physical addressing as needed.  There is no percentage limit of RAM imposed on running processes.

 

The fact that I see CTD's playing Skyrim on my system (64-bit Win7 Enterprise on a Core i5 with 32GB RAM) means it isn't a system architecture limitation.  Understanding that TESV.exe is a 32-bit executable, the problem is very likely due to a memory allocation call being out-of-bounds in certain situations, but without knowing the compiler settings used to create the EXE file, there's no way for me to know for sure.

 

Also, the location of any file on your hard drive has no bearing on how Windows treats the memory management for associated processes.  Moving your Steam folder out of "Program Files" simply alleviates issues that arise from the Windows7 administrator permissions model because that directory and its contents are protected.

[phazer11]

Well put couldn't have said it better roland. As it's been said the problem seems to be that 32-bit executables (in this case Skyrim) can only use 4GB of RAM. Skyrim's RAM usage is a combination of mirrored VRAM, RAM used by the main engine to run the game, and then RAM used for any scripts you have running. Therefore this is likely a combined issue of a 32-bit program limitation hitting the design flaws of DirectX9.

[z929669]

Umm, no.  Please do not post authoritative-sounding treatises on computer memory architecture without knowing the material yourself.

 

Windows has never limited single programs to a fixed percentage of physical RAM.  In the era of 32-bit architectures, Windows NT and its progeny used a virtual memory architecture that presented every process executed with 4GB of addressable memory space.  In other words, each program believed it was running by itself on a machine with 4GB of RAM installed.  How the virtual addresses mapped to the physical memory was handled by the Windows memory manager, and it often involved using hard-disk space as temporary storage (the paging file).  As the concurrent running processes' memory demands approached the limit of installed RAM, Windows would attempt to swap more and more memory data to the paging file, causing thrashing and eventually out-of-memory errors that resulted in process crashes or system lockups.

 

The 4GB "limit" existed because that's the maximum range that could be described (or addressed) using 32 bits when each unique combination of bits references a unique byte of memory.  [2 raised to the 32nd power is 4,294,967,296]

 

In the mid-'90s, the Physical Address Extension (PAE) architecture enhancement was developed that enabled CPUs to use 36+ bits for addressing physical RAM, up to 64GB...however, the Windows virtual memory addressing scheme remained at 32-bits, meaning each process could "see" a maximum of 4GB.

 

In reality, each system required dedication of some memory for video addressing, memory-mapped I/O, and other system tasks, effectively limiting 32-bit personal computers to 3.0 - 3.5GB of usable physical RAM.  Microsoft also introduced an artificial limit in their non-server 32-bit Windows operating system code that locks out addressing above 4GB.  (See "3GB Barrier" for more detail.)

 

In 64-bit land, none of this matters.  CPUs enter "long mode" and can address a theoretical maximum 16-exabytes of RAM when using all 64 bits, and the virtual process space in 64-bit Windows uses 48 bits for addressing, allowing each process to "see" 256 terabytes of available memory, while continuing to use mapping tables and paging file where needed to translate to physical addressing as needed.  There is no percentage limit of RAM imposed on running processes.

 

The fact that I see CTD's playing Skyrim on my system (64-bit Win7 Enterprise on a Core i5 with 32GB RAM) means it isn't a system architecture limitation.  Understanding that TESV.exe is a 32-bit executable, the problem is very likely due to a memory allocation call being out-of-bounds in certain situations, but without knowing the compiler settings used to create the EXE file, there's no way for me to know for sure.

 

Also, the location of any file on your hard drive has no bearing on how Windows treats the memory management for associated processes.  Moving your Steam folder out of "Program Files" simply alleviates issues that arise from the Windows7 administrator permissions model because that directory and its contents are protected.

 

Great post. Thanks.

 

Could you expound a bit on this? The EXE is inherently limited by the amount of memory that it itself can address, is it not? Are you referring to a call made by the EXE or a call made by WMM?

 

Is it likely that a pattern of crashes at or near 2.5-ish GB may be related to LAA in a 32-bit Windows env?

 

Would you care to contribute some wisdom or at least content-edit the article we are preparing on the wiki?