CTD at 3.1GB

[viking]

Some key posts on this and related threads (experts feel free to note any errors or insights):

• Post #11
• Post #27
• Post #40 (this is where the 500/430 ratio seems to have originated.)
• Post #66
• Post#278
• Post #313
• Post #1 (adjacent thread: Caution about memory optimisers)
• Post #64 (adjacent thread: Caution about memory optimisers)
• Post #68 (adjacent thread: Caution about memory optimisers)
• Post #70 (adjacent thread: Caution about memory optimisers)

Wiki article (draft)

 

Thanks,

STEP

 

OP follows

---

First of all I wanted to thank you all for the great work you have done with STEP. Skyrim is the first game I installed on this computer and you guys have made it an AMAZING game. That being said, I have an issue that I hope you can help me solve.

 

My setup:

Vanilla Skyrim

gtx670 w/ 4GB @1080p w/ latest driver

16GB Memory

3770K at about 4GHz

Windows 8 64bit

ENB 149

Ultra settings

Highest available texture/quality

Mod Organizer

Step 2.2.1 + Skyrim Revisited + others

 

I have noticed a post here and there saying that Skyrim can't really address more than about 3.1GB of memory without issues. This seems to jive with my experience, meaning I CTD every time my memory hits that mark, but I couldn't really find anything definitive on the topic. The issue with googling the topic is the pre 1.3 skyrim that couldn't address more than 2gb of memory.

 

My mod list is mostly based on STEP which is why I came here for help, with about ten mods added onto the end (Interesting NPCs, Detailed Cities, Economics, COT, and a couple others). The reason I haven't included my mod list is that it doesn't seem to matter. As long as I keep the memory usage below 3GB I can have pretty much any combination of mods.

 

What I have tried so far (in no particular order):

• resetting ini files
• removing enb
• not using attklt
• only using a new game
• removing all mods and adding one by one until issue crops up
• running as admin
• watching the papyrus log - it seems relatively clean, no obvious errors right before CTD

Yes, I can run STEP just fine without any issues, but I also never get near 3GB of memory. I have tracked VRAM usage as well and have seen a max of 2.7GB/4GB.

 

As an example of where I might run into issues: I start a new character with Alternate Start. I start with Breezehome. Run out of Whiterun, past the Brewry, up the hill to the bandits. Enter the cave (watching memory usage with Elys MemInfo), and it dies right after I see 3GB. I have this same issue not using AS, sitting through the intro, and then running over to whiterun.

 

I'm sorry if this post is all over the place. I have spent more than a week trying to solve this issue, and the only solution I have found is to reduce memory usage. I have got to the point where I can exchange two texture packs and get into the cave without a CTD, but with both I get a CTD. I didn't even think texture packs should even affect CTDs, but I'm relatively new to Skyrim on the PC, so I could be wrong. I also found I could get a bit further with ENB turned off, but would still crash once I got above 3GB of memory. Finally, if I reload a game after a CTD, I can play just fine...until I reach 3GB of memory.

 

I really hope you guys can help. I more than willing to try anything at this point, besides just disabling all of the mods.

[Besidilo]

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory.

 

Google "Large Address Aware" if you don't believe me.

 

EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim...

 

https://www.gamasutra.com/view/feature/132003/sponsored_feature_ram_vram_and_.php?print=1

 

https://www.techpowerup.com/forums/showthread.php?t=91260

[Salvador]

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory.

 

Google "Large Address Aware" if you don't believe me.

 

EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim...

 

https://www.gamasutra.com/view/feature/132003/sponsored_feature_ram_vram_and_.php?print=1

 

https://www.techpowerup.com/forums/showthread.php?t=91260

 

Really,

 

That is very interesting indeed.... but also worrisome. Because quite a bit of the VRAM is mirrored in RAM. This would suggest that DDSopting would have double the effect?... Unless Z is absolutely right and there is no mirroring going on. Nonetheless, DDSopt seems to be really important now.

 

@ Viking Yes my bad, it should only page when running out of RAM, however, OS dumps quite some things in Page (that are not needed soon) to free up RAM. Processes can however address page file directly. Still, the process should be able to address 4GB of space with a 32 bit integer.

 

Besidilo's post would suggest that the other 1GB is in VRAM. Isn't VRAM usage addressed by the graphics driver? 

[viking]

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

[Besidilo]

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

 

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

 

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

 

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

 

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.

[MontyMM]

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

 

I have not been formally recording the results, but since I returned to Skyrim a few weeks ago I've been monitoring it's behaviour.  In my case there is no doubt - adding textures does increase both VRAM & RAM usage in a very close ratio.  At the moment, my Skyrim setup is in bits, but I will do some proper benchmarks at a later date.

 

As Viking is at pains to point out, there is something significant going on here, but so far we know very little.  One fundamental point is that correlation does not prove causation - although the crashes appear to occur at a certain level of memory utilisation, it is possible that other less visible demands increase in tandem with memory usage, and these could be implicated. 

 

That said, I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.

 

[viking]

The allocated virtual memory limit for applications with LAA patch under x86 is 4GB. Bear in mind, this is not just system RAM, it's total memory used (not pre-cached), so system memory + video memory.

 

Google "Large Address Aware" if you don't believe me.

 

EDIT: from what I'm reading right now, this brings me to question the logic behind getting a 4GB VRAM card specifically for Skyrim...

 

https://www.gamasutra.com/view/feature/132003/sponsored_feature_ram_vram_and_.php?print=1

 

https://www.techpowerup.com/forums/showthread.php?t=91260

Alright. I finally had a chance to read through both of these. Up front, the gamasutra article is quite misleading and I wouldn't recommend it. The links it references are all high quality and the author did know what he was talking about (for the most part), but leaves out some essential bits if you aren't familiar with the area. The second link is accurate, but somewhat dated. I don't really feel like going into more detail here, but can if someone is interested.

 

In response to your comment, we still are not sure exactly how Skyrim manages its vram. A link earlier in this thread (and actually in your first link) discusses the windows patch that helps manage vram outside of the address space of the program. In versions of directx earlier than 10, the vram is mirrored in the application address space, but I haven't seen any clear indications of skyrim being strictly directx 10 or strictly directx 9 (please link if you can), and so, in my opinion, we are unable to answer with any certainty if all vram is always mirrored in application ram for skyrim. 

 

Note that this is a different issue than if vram on the graphics card is in the same address space as the application. Based on my experience, this is not the case. My skyrim ram + skyrim vram often exceed 4gb.

 

As for if you should get a 4GB graphics card, I often go above 2GB of VRAM, but have yet to hit 3GB vram. I think you will still find an advantage of having a 3GB+ graphics card.

[viking]

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

 

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

 

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

 

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

 

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.

It is a shame that we keep posting at the same time. I assume that your second paragraph means that virtual memory is different than physical memory, and in that we are in agreement.

 

I'm not sure why you think that you can't measure VRAM usage accurately. I agree it is difficult to pin VRAM usage to a particular application, but you can get pretty good idea.

 

I addressed your other points in my previous post.

[viking]

That said I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.

I think that this is a very important point. Assuming memory is the issue (which can't do with complete certainty), the issue very easily could be a limitation of the CreationKit engine (is that what it's called?).

[Salvador]

Besidilo: I have not yet read the articles that you linked to, but they look very interesting. I also am not sure what you mean by "pre-cached" memory, but would be interested if you could explain more. Your comment that LAA applications are limited to 4GB for System Ram + VRam, however, doesn't line up with what I have seen. I often see my VRAM at about 2.7 GB and my Ram at about 3.1 GB...which is obviously more than 4GB. The only way I can reconcile your comment with my experience is if the VRAM that is mirrored in RAM is not counted twice. That is certainly a possibility. I apologize if this is addressed in your articles, and feel free to respond to that effect if that is the case. 

 

I would like to point out for Salvador's sake, and for everyone else reading this thread, that we have done only very limited tests about the mirroring of VRAM and RAM. It is not outside the realm of possibility that the VRAM is temporarily loaded into memory before being forwarded to VRAM...or something to that effect. We also don't know for sure if the VRAM is considered as part of the 4GB address space. All we know for certain at this point is that texture files in skyrim DO affect VRAM usage AND Ram usage. I'll update this comment once I have a chance to read Besidilo's links, they look to have a lot of good information in them.

I shouldn't have used the term "pre-cached" since caching memory already implies a certain buffer.

 

My point was to not get confused by the extra memory that would be cached in any case, any application may appear to use more memory than it actually needs, up until the point it runs out completely.

 

You can't measure VRAM usage accurately, and that has been addressed a lot of times already. You're probably thinking about the extra textures that has been buffered to the video RAM, but aren't actually used by the engine.

 

With regards to the memory limitation, Dx9 mirrors VRAM onto the system memory, in turn reducing your actual system RAM limit by the amount of VRAM in use for any process. Microsoft got rid of this limitation in DirectX 10. x86 processes with Large Address Aware patch can use up to 4GB total system memory.

 

In other words, it's complicated. But it can be assumed that a crash would occur when Skyrim with LAA suddenly requires more than 4GB of RAM+VRAM. How to measure that point accurately is another matter. At least that's how I understand it.

 

DX9 is partially fixed, and not fully mirrored anymore, but the allocated VRAM is (the part allocated to Skyrim (textures)) according to besidilo site link (which is really good, love the references). In short, for DX9 the allocated VRAM grows (and is duplicated in RAM) when more VRAM is needed by the game engine and thus more memory is used when more VRAM is allocated (compressed textures I presume).

 

My understanding from your words, and from quickly reading through the articles is that you should also count the VRAM, at least the part of textures that is loading into VRAM. Not all the AA and AF and other processing going on in your graphics card that is also loaded into VRAM. So, from those 2.7GB only, lets say (just random example), a third are actually loaded textures, the remainder is used and allocated by the driver. However, I was under the impression that the driver handles VRAM allocation completely, which apparently is incorrect, or perhaps the driver sort of functions as the OS does for RAM when it comes to allocating VRAM for Skyrim.

 

I know only limited tests are done. I never said everything or nothing is mirrored. In the past things were completely mirrored as can be understood from the literature such as besidilo points to. And now this is mostly fixed, apart from DX9 which is only partially fixed. However this does not explain how it comes you can continue to play when VRAM cap is reached. This would suggest that dynamic or system VRAM, is RAM allocated as being VRAM? If that is indeed the case, than that dynamic VRAM (if allocated to Skyrim engine and not the VGA driver) should also be duplicated. This would suggest that above VRAM cap your memory usage grows rather fast. Would be nice to see full RAM usage on Zs benchmarks. You'd expect that an increase in system VRAM of 200mb means a increase in memory usage of 400mb, could perhaps be a nice test to measure if there is mirroring going on.

 

Love this thread and these type of discussions. Apologies to viking for changing the topic quite a bit.

[Salvador]

I see I'm slow with posting.

 

@Viking

 

As I liked the gamasutra link, due to the good references, I would love to know what is so misleading about it. It did indeed mis the info in the earlier link that showed that for DX9 only allocated VRAM is mirrored and for DX10 and 11 nothing is mirrored in RAM anymore (after fix).

 

I believed Skyrim was DX9, but apparently it could be DX10 as well? both XBOX and playstation are DX9 only right? Or is PC version considerably different?

[Besidilo]

VRAM usage is almost never fixed. It's dynamic + memory buffer.

 

Which is why it's pretty much impossible to measure it easily or accurately for any game. No, MSI Afterburner or GPU-Z won't do the job, as they will only show you the allocated memory, with no regards as to what is actually being buffered to the memory, how much is used by the application, how much by background process or system UI.

 

Oh, and it's Besi·dilo.

 

And no, Skyrim isn't "partially DirectX 9". It's fully DirectX 9 and there are no Dx10/11 executables. So the normal memory limitations apply.

[Besidilo]

 

That said I think memory is certainly a suspect.  In the opinion of ENB Boris, many games suffer from bugs with addressing memory above 2GB.  I think this is worth considering, since it is possible that (assuming memory is the problem), it could be an implementation problem rather than a fundamental limitation, which would tally with the unexpected and varying limiting figure of 3 - 3.5GB usage.

I think that this is a very important point. Assuming memory is the issue (which can't do with complete certainty), the issue very easily could be a limitation of the CreationKit engine (is that what it's called?).

 

The engine is called Creation Engine.

 

https://elderscrolls.wikia.com/wiki/The_Creation_Engine

https://www.rockpapershotgun.com/2011/01/18/skyrims-creation-engine-detailed/

[Salvador]

VRAM usage is almost never fixed. It's dynamic + memory buffer.

 

Which is why it's pretty much impossible to measure it easily or accurately for any game. No, MSI Afterburner or GPU-Z won't do the job, as they will only show you the allocated memory, with no regards as to what is actually being buffered to the memory, how much is used by the application, how much by background process or system UI.

 

Oh, and it's Besi·dilo.

 

And no, Skyrim isn't "partially DirectX 9". It's fully DirectX 9 and there are no Dx10/11 executables. So the normal memory limitations apply.

Hmmm, how did I got your name wrong?.. apologies.

 

I believed it was DX9 only, yet Viking said it could partially be DX10. If DX9 only this means that all allocated VRAM to skyrim is mirrored in system memory.

Yes I know it is all allocated and not necessarily used. But this should suggest, that any increase in dynamic VRAM (or system VRAM) is mirrored in RAM. Therefore, by just increasing texture sizes when you reached your VRAM cap, you could perhaps get an idea on how much is really mirrored (allocated. The problem occurs with 3.1GB allocated memory).

 

However, I need to find the link, in original windows 7 RAM allocated by sytem was shown, and because of the improved pre-fetch, it had very high memory usage numbers, which upset quite a lot of people, so they changed that. So even the allocation numbers are somewhat incorrect.

 

Edit: fixed names

[TechAngel85]

Looks like I sparked up quite a discussion! :D

 

It only popped up in my head last night... "hasn't any thought of the RAM limitation with x86 architecture?"

 

It just made since in my mind that any programs that passes this limit and was build on this architecture is going to have issues. Since Skyrim was build essentially with Windows XP era technology (DX9, etc) then this limitation should also apply to Skyrim. Keeping that logic in mind, if Skyrim processes are trying to allocation more than 4GB total memory, you could expect issues to arise. (CTD, freezing, etc).

 

EDIT:

Who marked this thread as solved because the discussion is clearly continuing and unclear of what is causing the real issue. I'm taking it off.

 

EDIT: I know see Viking did it in the title.

@ Viking,

The admins or moderators (myself) will mark the thread as solved when the issues have clearly be resolved. We have a special "pre-tag?" for this so there is no need to edit the title to mimic it.

[viking]

I'm going to try to get the thread back on track:

 

Besidilo, I'm not sure who got your name wrong. If it was me, sorry about that. I know very little about VRAM. A link with more info would be more helpful. The more we understand about the behavior of VRAM the more likely we are to understand this issue. I read in some forum somewhere (highly reliable...) that Skyrim was partially directx10. Don't trust me on that. It makes far more sense for it to be directx9, especially since xbox is limited to that.

 

Salvador, I agree that the references is Besidilo's links were excellent. I just had a hard time following the first one. I didn't notice anything incorrect in the links, there were just a few times when things like the maximum ram in a system was taken to be a constant across all systems. Again, the information was correct, just difficult to follow and left some key pieces of information out IMO. 

 

z recommended several test we can run to test the memory/vram correlation, and I will run them this weekend. Are there any other ideas on how we can get to the root of this issue?