CTD at 3.1GB

[viking]

Some key posts on this and related threads (experts feel free to note any errors or insights):

• Post #11
• Post #27
• Post #40 (this is where the 500/430 ratio seems to have originated.)
• Post #66
• Post#278
• Post #313
• Post #1 (adjacent thread: Caution about memory optimisers)
• Post #64 (adjacent thread: Caution about memory optimisers)
• Post #68 (adjacent thread: Caution about memory optimisers)
• Post #70 (adjacent thread: Caution about memory optimisers)

Wiki article (draft)

 

Thanks,

STEP

 

OP follows

---

First of all I wanted to thank you all for the great work you have done with STEP. Skyrim is the first game I installed on this computer and you guys have made it an AMAZING game. That being said, I have an issue that I hope you can help me solve.

 

My setup:

Vanilla Skyrim

gtx670 w/ 4GB @1080p w/ latest driver

16GB Memory

3770K at about 4GHz

Windows 8 64bit

ENB 149

Ultra settings

Highest available texture/quality

Mod Organizer

Step 2.2.1 + Skyrim Revisited + others

 

I have noticed a post here and there saying that Skyrim can't really address more than about 3.1GB of memory without issues. This seems to jive with my experience, meaning I CTD every time my memory hits that mark, but I couldn't really find anything definitive on the topic. The issue with googling the topic is the pre 1.3 skyrim that couldn't address more than 2gb of memory.

 

My mod list is mostly based on STEP which is why I came here for help, with about ten mods added onto the end (Interesting NPCs, Detailed Cities, Economics, COT, and a couple others). The reason I haven't included my mod list is that it doesn't seem to matter. As long as I keep the memory usage below 3GB I can have pretty much any combination of mods.

 

What I have tried so far (in no particular order):

• resetting ini files
• removing enb
• not using attklt
• only using a new game
• removing all mods and adding one by one until issue crops up
• running as admin
• watching the papyrus log - it seems relatively clean, no obvious errors right before CTD

Yes, I can run STEP just fine without any issues, but I also never get near 3GB of memory. I have tracked VRAM usage as well and have seen a max of 2.7GB/4GB.

 

As an example of where I might run into issues: I start a new character with Alternate Start. I start with Breezehome. Run out of Whiterun, past the Brewry, up the hill to the bandits. Enter the cave (watching memory usage with Elys MemInfo), and it dies right after I see 3GB. I have this same issue not using AS, sitting through the intro, and then running over to whiterun.

 

I'm sorry if this post is all over the place. I have spent more than a week trying to solve this issue, and the only solution I have found is to reduce memory usage. I have got to the point where I can exchange two texture packs and get into the cave without a CTD, but with both I get a CTD. I didn't even think texture packs should even affect CTDs, but I'm relatively new to Skyrim on the PC, so I could be wrong. I also found I could get a bit further with ENB turned off, but would still crash once I got above 3GB of memory. Finally, if I reload a game after a CTD, I can play just fine...until I reach 3GB of memory.

 

I really hope you guys can help. I more than willing to try anything at this point, besides just disabling all of the mods.

[Neovalen]

Yeah problem is I'm not golden. Performance is perfect but after ~40 minutes... ram is done and ctd.

[JudgmentJay]

Such is the life of heavy Skyrim modding :(

[viking]

That's alot of mods, but I think I have more. Honestly I don't think textures are your problem. As you can see my VRAM usage is more than yours, but my RAM usage is considerably lower... over 1GB lower. That's huge. Personally, I would start with a fresh vanilla Skyrim and install mods a little at a time and monitor RAM usage to find out which ones add the most. I recently went through my list and got rid of the ones that I figured I could live without in an attempt to make my game stable with uGrids=7. Unfortunately that proved futile. The engine just can't handle it.

 

I am aware that my count of mods is not that high, but some of them use a ridiculous amount of memory. I wasn't certain before starting this thread that there is a (relatively) hard 3.1 GB ram limit, but now that it seems pretty certain, I may go back and do just what you suggest. I think I will still gain a bit from ddsopting the textures I've got and will start with that, because, unfortunately, I have already cut my non-texture mods just about to a minimum for me.

[z929669]

Reading through this thread, I am not convinced that this is a RAM issue at all (and I echo Neovalen's JudgmentJay's comment about system RAM allotted to TEXV.exe being higher than 1.8 GB). Something strange is happening if people are getting system RAM usage that high from this single process.

 

Are we certain about these RAM allocations being specifically tied to TESV.exe?

 

Skyrim does not allocate much VRAM to RAM until it runs out of VRAM. Even then, it is not so much that it could additively boost my Skyrim RAM use to more than 3 GB. My benchmarks are pretty detailed in this regard.

 

Furthermore, is it a fact that this "bleed over" VRAM is in fact charged against the TESV.exe-allocated RAM usage?

 

Sounds like ATTK could help out here (not), regardless, but I would also not rule out a setup issue at this point. I have had 300+ mods installed in Morrwind, Oblivion and Skyrim, and I have never had CTDs related to RAM caps ... they were always related to setup issues.

 

(also, try using Process Hacker. It is even better than Process Explorer)

[JudgmentJay]

Skyrim does not allocate much VRAM to RAM until it runs out of VRAM.

That's certainly not my experience.

[z929669]

Skyrim does not allocate much VRAM to RAM until it runs out of VRAM.

That's certainly not my experience.

The proof is in the pudding if you look at the benchmarks I point to up there.

 

Would anybody getting these huge spikes mind replicating a couple of these benchmarks on their systems? I'd like to see the diffs of Vanilla versus bench after section G versus bench after section M using the intro sequence. This will be more solid evidence of the dynamics between VRAM and RAM on your systems. Others have similar benchmarks on their user pages (see Kelmych and techangel85), and I have not seen anything that is inconsistent with my own... yet.

[JudgmentJay]

I'm not going to go through all the trouble of doing a spreadsheet, but I can show the difference between vanilla and vanilla+TPC as far as VRAM/RAM usage goes. Unfortunately GPU-z doesn't seem to support dynamic memory monitoring on GTX680s. What could I use instead?

[viking]

Reading through this thread I am not convinced that this is a RAM issue at all (and I echo Neovalen's comment about system RAM allotted to TEXV.exe being higher than 1.8 GB). Something strange is happening if people are getting system RAM usage that high from this single process.

I'm not sure which of Neovalen's comments you are referring to. Also I'm not sure why it would be strange for this single process to use a lot of ram. I would be very happy if this is not a ram issue.

 

Are we certain about these RAM allocations being specifically tied to TESV.exe?

Yes? I'm not sure where you are coming from. I tracked the RAM (not VRAM) usage using Process Explorer (which you are familiar with); it was tracking only TESV.exe. VRAM, which was system wide, was tracked using GPUz.

 

Skyrim does not allocate much VRAM to RAM until it runs out of VRAM. Even then it is not so much that it could additively boost my Skyrim RAM use to more than 3 GB. My benchmarks are pretty detailed in this regard.

I haven't had a chance to read through your benchmark, but I will. I am more than happy to do any tests you are interested in, because I would much rather find a setup issue than reduce mod count. Unfortunately I won't have a chance until this weekend to do any real tests (meaning redoing step from scratch). I'd like to point out that my numbers, and all of my experiences with ctds, were all using attklt.

 

As for whether the "bleed over", as you called it, goes into the same address space, it would seem that it does; however we have no real "proof" beyond some anecdotal evidence and the microsoft directx patch details linked to earlier in this thread.

[JudgmentJay]

Did a run from Helgen cave to Riverwood. Not the most accurate thing in the world, but it correctly depicts what I've found in my testing.

 

Vanilla:

 

Vanilla+TPC:

 

So with ONLY Texture Pack Combiner being added, I see an increase of ~500MB VRAM and 430MB physical RAM.

[Neovalen]

Did a run trying to watch the Process Hacker... private memory got over 3gb and then as expected... boom! As with the others... VRAM / RAM seemed to be pretty much 1:1 (give or take slightly).

[TechAngel85]

I would expect this behavior at closer to 4GB because of the game being 86x architecture (4GB limitations), but not 3GB. However, searching on Google for a bit revealed similar cases on 86x architecture with failures around 3.5GB. This 86x architecture limitation could very well be the issue unless it's been ruled out...I haven't read the whole thread.

[Salvador]

Lets say Skyrim engine has a 3GB limit in mem. If it reaches the limit it should ask windows to allocate page memory.

Or is this limit intrinsic in the engine and therefore no page memory is allocated? I find all this rather strange, even very old 32bit programs can use a lot of mem, but simply load the overhead in page. Take an old firefox for example, and load as much as you can. Page file becomes useful I can promise. Same goes with 32bit photoshop or autocad etc. Now I know a game engine is different, but there must be something wrong within the engine itself if it fails to address more than 3GB of memory. I disable my page file on disk, this means page will be allocated in RAM instead! Since I have plenty of RAM I don't care, however, if you have limited RAM this is naturally not a good idea. I am curious now, if you disable your page file, does ram allocation change at all? will you still crash at 3GB? Does Skyrim actually get Page mem allocated? What happens with page mem usage?

 

If it is a engine issue, it can only be fixed by Bethesda.

 

I am curious as of what causes the serious hard cap at 3GB... Even 32 bit program ought to be able to reach ~3.5GB..

[TechAngel85]

Lets say Skyrim engine has a 3GB limit in mem. If it reaches the limit it should ask windows to allocate page memory.

Or is this limit intrinsic in the engine and therefore no page memory is allocated? I find all this rather strange, even very old 32bit programs can use a lot of mem, but simply load the overhead in page. Take an old firefox for example, and load as much as you can. Page file becomes useful I can promise. Same goes with 32bit photoshop or autocad etc. Now I know a game engine is different, but there must be something wrong within the engine itself if it fails to address more than 3GB of memory. I disable my page file on disk, this means page will be allocated in RAM instead! Since I have plenty of RAM I don't care, however, if you have limited RAM this is naturally not a good idea. I am curious now, if you disable your page file, does ram allocation change at all? will you still crash at 3GB? Does Skyrim actually get Page mem allocated? What happens with page mem usage?

 

If it is a engine issue, it can only be fixed by Bethesda.

 

I am curious as of what causes the serious hard cap at 3GB... Even 32 bit program ought to be able to reach ~3.5GB..

All good questions, but I would expect a game to crash or freeze if it had to start using page memory. VRAM is fast. RAM is much slower and causes stuttering in-game. However, page memory is extremely slow (especially on HDD)...slow enough that if RAM causes stuttering, a crash or freeze wouldn't come as a surprise for a game that called for something that was stored in page memory.

[Salvador]

Lets say Skyrim engine has a 3GB limit in mem. If it reaches the limit it should ask windows to allocate page memory.

Or is this limit intrinsic in the engine and therefore no page memory is allocated? I find all this rather strange, even very old 32bit programs can use a lot of mem, but simply load the overhead in page. Take an old firefox for example, and load as much as you can. Page file becomes useful I can promise. Same goes with 32bit photoshop or autocad etc. Now I know a game engine is different, but there must be something wrong within the engine itself if it fails to address more than 3GB of memory. I disable my page file on disk, this means page will be allocated in RAM instead! Since I have plenty of RAM I don't care, however, if you have limited RAM this is naturally not a good idea. I am curious now, if you disable your page file, does ram allocation change at all? will you still crash at 3GB? Does Skyrim actually get Page mem allocated? What happens with page mem usage?

 

If it is a engine issue, it can only be fixed by Bethesda.

 

I am curious as of what causes the serious hard cap at 3GB... Even 32 bit program ought to be able to reach ~3.5GB..

All good questions, but I would expect a game to crash or freeze if it had to start using page memory. VRAM is fast. RAM is much slower and causes stuttering in-game. However, page memory is extremely slow (especially on HDD)...slow enough that if RAM causes stuttering, a crash or freeze wouldn't come as a surprise for a game that called for something that was stored in page memory.

Yes exactly, that is why I disable page on disk. It automatically allocates in RAM instead (However, you should only do this if you have plenty, and never run out of RAM). Together with page any large address aware 32 bit process should certainly be able to reach the 4GB on 64bit OS. VRAM is not limited for most people having issues on this thread. Their VRAM cap is not reached.

Does Skyim use contiguous memory, or works in chunks?

 

A different question. DLLs usually allocate quite a bit of memory. Does having ENB DLL present or not make any difference on mem usage, just disabling ENB does not stop it from loading into mem. Might also be interesting to see what the different DLLs do. I find it strange to see these discrepancies between people in the amount of RAM usage, with similar STEP installments.....

[viking]

I only have a minute, but I wanted to clear something up before this thread gets too off track. In reference to System Memory (not VRAM): Memory is only paged out to disk if the entirety of physical memory is used up. This is not managed by individual processes, thus this cannot be a limitation of the game engine and is not related to the amount of memory given to a process. To the process, all it sees is the virtual memory addresses. When the process wants to get something from memory, it tells the OS it wants to get (virtual) address, lets say, 57. The OS then translates that into either the physical memory location or the page location on disk.

 

Now on to some of the comments: Disabling paging is a bad idea. It forces, as Salvador points out, all of the memory to be in physical memory and not paged to disk. Unfortunately, if the system runs out of physical memory it simply crashes. No ifs, ands or buts. It won't speed up a program because the OS already is trying to keep everything in memory that it possibly can. On a more technical note, some OSes may page things out to disk before it is necessary; however it keeps it in memory too. This just allows the OS to not have to copy a page from memory to disk in an "emergency".

 

I don't know as much about VRAM usage. From z's post (sorry! I forgot the number's that follow!) it seems that when VRAM runs out, the system allocates some graphics memory in RAM as well. This will increase stuttering, but not cause CTDs. This is described in much more detail in z's performance testing. If the system got to the point where both VRAM and Memory were paged to disk...you're in trouble. If somehow the system had to allocate some page space on disk for VRAM, your game would be so slow it would be unplayable. Don't do it. Buy more ram.

 

Salvador, you mention that this might be a Bethesda issue. That is currently one of the two likely possibilities, and the one we had agreed upon until z came along and suggested the possibility of a setup issue. Not bashing on z, I appreciate help from anyone who can give it.

 

Salvador, you also mentioned the issue of DLLs. You are right, DLLs are allocated in the same address space so they will take up some of the memory. I have found, though I hadn't previously mentioned, that removing ENB does reduce memory usage, but only slightly. I still have the same 3.1GB limit in that case.

 

Also, I don't think our STEP installments are all that similar, though no one else has posted their mod list. Mine, in particular, is not really STEP for more than textures and fixes. In particular, I have several very RAM heavy mods that are not in STEP (nor do I recommend that they be) and it is unlikely that others here have them all installed. Neo, based on his incredible Skyrim Revisited page, has somewhat similar install to mine, but again varies in some very large mods (like Interesting NPCs).

 

I really do not mean to discourage discussion, I just wanted to make sure things stayed on track. I'm not an expert on this topic, by any means, and I am just as baffled by the 3.1GB cap as everyone else. I apologize in advance if I have been rude to anyone...I haven't had my coffee yet...