GUIDE [WIP] DDSopt & Texture Overhauls

[z929669]

Discussion thread:
DDSopt Guide by STEP
Wiki Link

---

 

GET DDSopt:
Github Pre-release versions
Official Nexus versions (select pre-release update 4)

[bitdman]

That theory is more or less correct as I understand VRAM. VRAM operates very similar to system memory really. In windows or linux OS you have a "swap" which performs the same action as memory does for VRAM. If you can keep the swapped portion small enough, it is usually ok to have VRAM full with a portion of the rest in main memory.

 

I think, however, that the exact way VRAM works is different from system programs. Most of the video memory stored IS pertinent to whatever is going to the game at the time. In a system program all of the most frequented accessed data, the system stack and the system heap are all in main memory and what tends to get pushed off to the hard disk is the content of files that are loaded in memory at the time. VRAM has a stack and a heap as well, but the contents of VRAM memory are cycles through more often and more frequently than is main memory due to the fact that video cards often do needs to deal with very large files nearly randomly. So VRAM and a half in memory would be very close to threshold for acceptable performance. I think VRAM and a tenth or so would probably be approaching the territory where frame rates and noticeably effected. I don't have any data to back this claim up. However I could direct you to some very good articles on graphics and graphics processing if you are curious and want to know more.

Can't they (meaning the graphic card co.) come up with a way of handling the memory externaly and make slots so we can puchase and add memory as needed? Just like a PC motherboard;)

[mothergoose729]

That theory is more or less correct as I understand VRAM. VRAM operates very similar to system memory really. In windows or linux OS you have a "swap" which performs the same action as memory does for VRAM. If you can keep the swapped portion small enough, it is usually ok to have VRAM full with a portion of the rest in main memory.

 

I think, however, that the exact way VRAM works is different from system programs. Most of the video memory stored IS pertinent to whatever is going to the game at the time. In a system program all of the most frequented accessed data, the system stack and the system heap are all in main memory and what tends to get pushed off to the hard disk is the content of files that are loaded in memory at the time. VRAM has a stack and a heap as well, but the contents of VRAM memory are cycles through more often and more frequently than is main memory due to the fact that video cards often do needs to deal with very large files nearly randomly. So VRAM and a half in memory would be very close to threshold for acceptable performance. I think VRAM and a tenth or so would probably be approaching the territory where frame rates and noticeably effected. I don't have any data to back this claim up. However I could direct you to some very good articles on graphics and graphics processing if you are curious and want to know more.

Can't they (meaning the graphic card co.) come up with a way of handling the memory externaly and make slots so we can puchase and add memory as needed? Just like a PC motherboard;)

Unfortunately no they cannot. They DRAM is tied too closely to the cards bus for that. All of the DRAM is hard wired into the PCB and soldered in. There used to be a time when CPU cache was modular as well, when the pentium pro came out they moved it on chip,I believe for performance reason (I want to say latency issues). I believe memory in graphics has the same issues, where the demand for performance makes designing a system bus for plug and play memory impossible or impractical. 

[freche]

Was planning on Opting the HR DLC textures, but the batch file I downloaded is not made for the these instructions: https://forum.step-project.com/showthread.php?tid=228&pid=12819#pid12819

 

Instructions says to make two Tweaks folders but the batch file is made to handle just one.

Is it possible to extract them into the same folder and if so in which order?

[z929669]

Was planning on Opting the HR DLC textures, but the batch file I downloaded is not made for the these instructions: https://forum.step-project.com/showthread.php?tid=228&pid=12819#pid12819

 

Instructions says to make two Tweaks folders but the batch file is made to handle just one.

Is it possible to extract them into the same folder and if so in which order?

Just look at the first few lines of the BAT file and set up the folder structure likewise, or you can simply rename accordingly;)

[freche]

These are the line I have in the HRDLC_Fix-START.bat file I got from your post.

subst r: ".\HRDLC1\textures"
subst w: ".\HRDLC2\textures"
subst t: ".\HRDLC_Tweaks\textures"
subst u: ".\HD_Texture_DLC_Fix\textures"

Can't really work with that and two Tweaks folders.

Would need to either extract both 3.2 and 3.1 beta to the same folder (and I don't know in what order)

or an batch setup that can handle one more subst designed for one of the HRDLC_Tweaks versions.

 

or am I just too tired and stupid atm?

[z929669]

These are the line I have in the HRDLC_Fix-START.bat file I got from your post.

subst r: ".\HRDLC1\textures"
subst w: ".\HRDLC2\textures"
subst t: ".\HRDLC_Tweaks\textures"
subst u: ".\HD_Texture_DLC_Fix\textures"

Can't really work with that and two Tweaks folders.

Would need to either extract both 3.2 and 3.1 beta to the same folder (and I don't know in what order)

or an batch setup that can handle one more subst designed for one of the HRDLC_Tweaks versions.

 

or am I just too tired and stupid atm?

OK, I follow you now. I updated the reference post to sync up with the BAT file. I believe that frihyland recommended that 3.1b textures were superior to 3.2 where there are conflicts; however, I suggest that you compare these conflicts using this utility to decide for yourself.

[freche]

Thanks for the help z.

 

Now I have run into another problem.

I have used the TPC mod and planned on Optimizing the Combined textures, however all files are blue (some setting is preventing them from being optimized).

I have used the settings from the OP and then I have tested with ticking and un-ticking things in Ignore & Settings.

 

I have tried with both pre5b and pre3 versions of DDSOpt

(I had no problem optimizing the Vanilla & HRDLC textures)

 

EDIT - SOLVED

Managed to solve it, though I think it's kinda strange.

If I in DDSOpt selected the Texture folder (TPC/Combined/Data/Textures) I got everything in blue.

If I selected the Data Folder (TPC/Combined/Data) then the same files appeared in green.

[stoppingby4now]

My theoretical assessment is based on a lot of real-world testing and info submitted by others. The point is that the actual perceived threshold imposed by the GPU VRAM capacity is larger than that threshold. Whether it is closer to 1.1x or 1.5x remains to be empirically determined, preferably on a number of different systems, both single and multi-GPU.

 

If you reference the performance data (note table footnotes to col headers), you will see the evidence of the actual memory-management behavior between dedicated and dynamic VRAM. System RAM interaction is the black box, but the data in the tables suggests that 1.1x is pretty safe, and I am stipulating that as one approaches 1.5x, stuttering will become more and more noticeable. I assume that we would see stuttering something less than half of the time 1.5x ... how much less depends upon the sophistication of VRAM/RAM memory management, which will depend on a lot of things.

 

Theories are interesting, but only real test data provides the answers ;)

 

In practical terms, as one exceeds 1.0x, the potential for stuttering or graphical anomalies will exist. There is no solid case that can be made in regards to such a number beyond 1.0. In relation to the tables you posted, the shared memory is very low, which is what you want. There is always going to be some small amount of overhead imposed by the graphical sub system. But in terms of VRAM being over utilized and having to start relying on system RAM, the actual point at which someone will notice any glitches will be dependent on their system, and how often texture data is having to be switched out (though the later will be the biggest impact as frequency of swapping will be directly related to performance loss). In my recent tests (with a 3GB card) my reported System Shared RAM never exceeds 22MB, and I never see a drop in Dedicated GPU memory, which tells me the card is holding on to the texture data even if it's not using it (because it has the room to do so). That's the best scenario, and it also shows that it is not managed the same as system memory.

 

In terms of the data in the performance data tables, suggesting that 1.1x is safe I feel is misleading (and even more so to suggest anything beyond that). The key piece of data that is missing is whether or not texture data is being swapped out between System RAM and VRAM during those tests, and at what frequency. I have yet to determine any way of detecting that, if one even exists.

[z929669]

In practical terms' date=' as one exceeds 1.0x, the potential for stuttering or graphical anomalies will exist. There is no solid case that can be made in regards to such a number beyond 1.0. In relation to the tables you posted, the shared memory is very low, which is what you want. There is always going to be some small amount of overhead imposed by the graphical sub system. But in terms of VRAM being over utilized and having to start relying on system RAM, the actual point at which someone will notice any glitches will be dependent on their system, and how often texture data is having to be switched out (though the later will be the biggest impact as frequency of swapping will be directly related to performance loss). In my recent tests (with a 3GB card) my reported System Shared RAM never exceeds 22MB, and I never see a drop in Dedicated GPU memory, which tells me the card is holding on to the texture data even if it's not using it (because it has the room to do so). That's the best scenario, and it also shows that it is not managed the same as system memory.[/quote']

This makes sense, but when VRAM does not a have the luxury of accommodating unused data, it is swapping what it needs with system RAM (i.e., dynamic VRAM). The fact that it always uses some proportion of dynamic VRAM (i.e., system RAM) must mean that there is always a small buffer established (to resolve any potential VRAM faults??). My dynamic VRAM for base windows is only 25MB (see top of table in Guide), so this fits well with you only seeing ~22MB on a 3GB card.

In terms of the data in the performance data tables, suggesting that 1.1x is safe I feel is misleading (and even more so to suggest anything beyond that).

I disagree that it is misleading, because it is a fact that I get no stuttering on my system even with the 2k max overhauls and then some. It only happens when I pile on more demand, which means that the 1GB limit on my card(s) in my system is a soft threshold, so I reason that there must be something going on regarding buffers and memory management. I simply devised a plausible theory to explain it, and until an expert on this particular interaction between VRAM and system RAM can explain the reality in lay terms, I am sticking to it (although, I agree that 1.5x is probably off the mark ... 1.1x or simply "soft threshold" might be more realistic .... but still inexplicable).

The key piece of data that is missing is whether or not texture data is being swapped out between System RAM and VRAM during those tests, and at what frequency. I have yet to determine any way of detecting that, if one even exists.

Exactly my point. I'd really like to get a good explanation without digressing into the conceptual barriers of theoretical physics. I'd like a simple model (which is always possible, given an expert with enough creativity to extrapolate).

[stoppingby4now]

In practical terms' date=' as one exceeds 1.0x' date=' the potential for stuttering or graphical anomalies will exist. There is no solid case that can be made in regards to such a number beyond 1.0. In relation to the tables you posted, the shared memory is very low, which is what you want. There is always going to be some small amount of overhead imposed by the graphical sub system. But in terms of VRAM being over utilized and having to start relying on system RAM, the actual point at which someone will notice any glitches will be dependent on their system, and how often texture data is having to be switched out (though the later will be the biggest impact as frequency of swapping will be directly related to performance loss). In my recent tests (with a 3GB card) my reported System Shared RAM never exceeds 22MB, and I never see a drop in Dedicated GPU memory, which tells me the card is holding on to the texture data even if it's not using it (because it has the room to do so). That's the best scenario, and it also shows that it is not managed the same as system memory.[/quote'']

This makes sense, but when VRAM does not a have the luxury of accommodating unused data, it is swapping what it needs with system RAM (i.e., dynamic VRAM). The fact that it always uses some proportion of dynamic VRAM (i.e., system RAM) must mean that there is always a small buffer established (to resolve any potential VRAM faults??). My dynamic VRAM for base windows is only 25MB (see top of table in Guide), so this fits well with you only seeing ~22MB on a 3GB card.

That is true that System RAM will be used to swap out data when VRAM gets full. But one thing you also have to realize with the reported data from GPU-z, is that all of those figures are global figures as well (combined usage for all processes).

In terms of the data in the performance data tables, suggesting that 1.1x is safe I feel is misleading (and even more so to suggest anything beyond that).

I disagree that it is misleading, because it is a fact that I get no stuttering on my system even with the 2k max overhauls and then some. It only happens when I pile on more demand, which means that the 1GB limit on my card(s) in my system is a soft threshold, so I reason that there must be something going on regarding buffers and memory management. I simply devised a plausible theory to explain it, and until an expert on this particular interaction between VRAM and system RAM can explain the reality in lay terms, I am sticking to it (although, I agree that 1.5x is probably off the mark ... 1.1x or simply "soft threshold" might be more realistic .... but still inexplicable).

The fact that you don't get stuttering in those limited tests doesn't mean anything conclusive. The total amount of GPU System RAM (or Dynamic RAM as GPU-z calls it) reported is extremely low. The key piece of data is Dedicated VRAM, which doesn't even hit your VRAM barrier in your tests. So again, without knowing if any texture data was being swapped, it's all conjecture and guessing.

 

Beyond data being swapped to System RAM to make room, the next real piece of data that we need is how often data is having to be swapped. Even if GPU System RAM is reported at 200MB, and Dedicated GPU RAM is topped out, that in no way is an indication of a soft limit for stutter free play. Again, you need to know the frequency at which the data is swapped. Even when the frequency of swapping increases, the results are going to vary from system to system based on factors such as System RAM type, RAM speed, number of GPU's, northbridge chip, etc.

 

That is why I feel it is misleading, and I still stick to that.

The key piece of data that is missing is whether or not texture data is being swapped out between System RAM and VRAM during those tests, and at what frequency. I have yet to determine any way of detecting that, if one even exists.

Exactly my point. I'd really like to get a good explanation without digressing into the conceptual barriers of theoretical physics. I'd like a simple model (which is always possible, given an expert with enough creativity to extrapolate).

We don't need physics, we just need a way to see/report on what is actually occurring during game play, specifically in regards to swapping between VRAM and System RAM. I have yet to find anything that can report that. The same concept applies in performance testing server deployments. Stress the system and gather performance data, including whether or not any heavy paging occurs. The same concept applies here, just a different sub-system. I wish Performance Monitor had sensors for the GPU.

[stoppingby4now]

Follow up to fruther expand on why I think the 1.X speculation is mis-leading.

 

At least in terms of any texture data that is pushed to System RAM, it may just be holding data there that it doesn't currently need (and may not need for quite some time) because room had to be made for the current scenes. There may even still be some data in VRAM that isn't being used, but it hasn't needed to swap out yet. I just can't buy any speculation of 1.X times combined GPU memory use to be considered safe based on the potential for the above scenario, particularly because any texture data currently being held in System RAM just in case, may not even be used again in the current session.

 

Once swapping has to occur, any use above 1.0 immediately has the potential to cause problems, and  my point is that the key is frequency (and in tern influenced by system spec).

[z929669]

In practical terms' date=' as one exceeds 1.0x' date=' the potential for stuttering or graphical anomalies will exist. There is no solid case that can be made in regards to such a number beyond 1.0. In relation to the tables you posted, the shared memory is very low, which is what you want. There is always going to be some small amount of overhead imposed by the graphical sub system. But in terms of VRAM being over utilized and having to start relying on system RAM, the actual point at which someone will notice any glitches will be dependent on their system, and how often texture data is having to be switched out (though the later will be the biggest impact as frequency of swapping will be directly related to performance loss). In my recent tests (with a 3GB card) my reported System Shared RAM never exceeds 22MB, and I never see a drop in Dedicated GPU memory, which tells me the card is holding on to the texture data even if it's not using it (because it has the room to do so). That's the best scenario, and it also shows that it is not managed the same as system memory.[/quote'']

This makes sense, but when VRAM does not a have the luxury of accommodating unused data, it is swapping what it needs with system RAM (i.e., dynamic VRAM). The fact that it always uses some proportion of dynamic VRAM (i.e., system RAM) must mean that there is always a small buffer established (to resolve any potential VRAM faults??). My dynamic VRAM for base windows is only 25MB (see top of table in Guide), so this fits well with you only seeing ~22MB on a 3GB card.

That is true that System RAM will be used to swap out data when VRAM gets full. But one thing you also have to realize with the reported data from GPU-z, is that all of those figures are global figures as well (combined usage for all processes).

Oh, I realize that ... this is the reason that you see the base windows stats at the top of that first table in the DDSopt Guide ;)

In terms of the data in the performance data tables' date=' suggesting that 1.1x is safe I feel is misleading (and even more so to suggest anything beyond that).[/quote']

I disagree that it is misleading' date=' because it is a fact that I get no stuttering on my system even with the 2k max overhauls and then some. It only happens when I pile on more demand, which means that the 1GB limit on my card(s) in my system is a soft threshold, so I reason that there must be something going on regarding buffers and memory management. I simply devised a plausible theory to explain it, and until an expert on this particular interaction between VRAM and system RAM can explain the reality in lay terms, I am sticking to it (although, I agree that 1.5x is probably off the mark ... 1.1x or simply "soft threshold" might be more realistic .... but still inexplicable).[/quote']

The fact that you don't get stuttering in those limited tests doesn't mean anything conclusive. The total amount of GPU System RAM (or Dynamic RAM as GPU-z calls it) reported is extremely low. The key piece of data is Dedicated VRAM, which doesn't even hit your VRAM barrier in your tests. So again, without knowing if any texture data was being swapped, it's all conjecture and guessing.

It actually gets very near that barrier in my tests, but it does not ever seem to break it ... and that is precicely my point. Dynamic VRAM increases at those upper limits faster than does dedicated ... just look at the Skyrim HD table and note how you get nearly a 200% increase in dynamic VRAM for only a fractional increase in dedicated VRAM once the latter exceeds around 960MB on my 1 GB card(s). The opposite is true up to that point, so I would say that there is a soft threshold that begins to buffer using system RAM beginning at about 96% of dedicated VRAM utilization. Just look at the relative change within and among those two columns. ... That is the crux of the biscuit.

Beyond data being swapped to System RAM to make room' date=' the next real piece of data that we need is how often data is having to be swapped. Even if GPU System RAM is reported at 200MB, and Dedicated GPU RAM is topped out, that in no way is an indication of a soft limit for stutter free play. Again, you need to know the frequency at which the data is swapped. Even when the frequency of swapping increases, the results are going to vary from system to system based on factors such as System RAM type, RAM speed, number of GPU's, northbridge chip, etc.

 

That is why I feel it is misleading, and I still stick to that.[/quote']

Actually, the GPU RAM that overflows out to system RAM will be dismally slow in any system (relative to dedicated VRAM), even uber. So I would argue that once the process demands any piece of overflow data in real time, that will cause stutter in any system (perhaps even very bad stutter). The key is in the MM algorithm and whether or not it is probability-based (and if so, how well). That is what I want to know about ...

The key piece of data that is missing is whether or not texture data is being swapped out between System RAM and VRAM during those tests' date=' and at what frequency. I have yet to determine any way of detecting that' date=' if one even exists.[/quote'']

Exactly my point. I'd really like to get a good explanation without digressing into the conceptual

barriers of theoretical physics. I'd like a simple model (which is always possible, given an expert with enough creativity to extrapolate).

We don't need physics, we just need a way to see/report on what is actually occurring during game play, specifically in regards to swapping between VRAM and System RAM. I have yet to find anything that can report that. The same concept applies in performance testing server deployments. Stress the system and gather performance data, including whether or not any heavy paging occurs. The same concept applies here, just a different sub-system. I wish Performance Monitor had sensors for the GPU.

Agreed, but it sounds like Process Explorer can help (and a bit of deductive reasoning)

[z929669]

Follow up to fruther expand on why I think the 1.X speculation is mis-leading.

 

At least in terms of any texture data that is pushed to System RAM, it may just be holding data there that it doesn't currently need (and may not need for quite some time) because room had to be made for the current scenes. There may even still be some data in VRAM that isn't being used, but it hasn't needed to swap out yet. I just can't buy any speculation of 1.X times combined GPU memory use to be considered safe based on the potential for the above scenario, particularly because any texture data currently being held in System RAM just in case, may not even be used again in the current session.

 

Once swapping has to occur, any use above 1.0 immediately has the potential to cause problems, and the my point is that the key is frequency (in term influenced by system spec).

See my original theory on the point of overflow and probability of need. Setting aside in system RAM is still much faster than reacquiring from the process (i.e., disk), and if the MM algorithm is well conceived, then the probabilities that a given piec of data will be needed at any given time should factor into the equation.

[stoppingby4now]

Follow up to fruther expand on why I think the 1.X speculation is mis-leading.

 

At least in terms of any texture data that is pushed to System RAM, it may just be holding data there that it doesn't currently need (and may not need for quite some time) because room had to be made for the current scenes. There may even still be some data in VRAM that isn't being used, but it hasn't needed to swap out yet. I just can't buy any speculation of 1.X times combined GPU memory use to be considered safe based on the potential for the above scenario, particularly because any texture data currently being held in System RAM just in case, may not even be used again in the current session.

 

Once swapping has to occur, any use above 1.0 immediately has the potential to cause problems, and the my point is that the key is frequency (in term influenced by system spec).

See my original theory on the point of overflow and probability of need. Setting aside in system RAM is still much faster than reacquiring from the process (i.e., disk), and if the MM algorithm is well conceived, then the probabilities that a given piec of data will be needed at any given time should factor into the equation.

You pretty much back my point. :P

 

As an example, back when we were evaluating Vista for clients, we initially received results that didn't make sense at first in terms of system memory usage. By all accounts, we were showing that we were on the brink of needing more memory to support the applications, which didn't make sense given that nothing had changed in code. The fact is, that Microsoft changed the memory sub-system in Vista to start pre-loading heavily used applications into memory (and naturally it tended to include their products even before first use), so reported use (nearly 100% memory utilization) wasn't realistic and we had to adjust.

 

The same concept should apply to the GPU as well. Hoard a texture in System RAM if it puts it there just in case, because it's faster to load than from disk. So, just because it is sitting in System RAM does not equate to it's requirement for use ever again in the current session.

 

My point is that speculating 1.X of overall GPU related memory use as being safe, without taking into consideration it's actual need, is misleading. I can only see a case for determining the frequency of swapping as being the only real measure of texture use to graphical anomalies. Consequently, a high frequency of swapping should line up with over-use of higher resolution textures, which would mean anything beyond 1.0 is immediately noticeable.

[z929669]

 

Follow up to fruther expand on why I think the 1.X speculation is mis-leading.

 

At least in terms of any texture data that is pushed to System RAM, it may just be holding data there that it doesn't currently need (and may not need for quite some time) because room had to be made for the current scenes. There may even still be some data in VRAM that isn't being used, but it hasn't needed to swap out yet. I just can't buy any speculation of 1.X times combined GPU memory use to be considered safe based on the potential for the above scenario, particularly because any texture data currently being held in System RAM just in case, may not even be used again in the current session.

 

Once swapping has to occur, any use above 1.0 immediately has the potential to cause problems, and the my point is that the key is frequency (in term influenced by system spec).

See my original theory on the point of overflow and probability of need. Setting aside in system RAM is still much faster than reacquiring from the process (i.e., disk), and if the MM algorithm is well conceived, then the probabilities that a given piec of data will be needed at any given time should factor into the equation.

You pretty much back my point. :P

 

As an example, back when we were evaluating Vista for clients, we initially received results that didn't make sense at first in terms of system memory usage. By all accounts, we were showing that we were on the brink of needing more memory to support the applications, which didn't make sense given that nothing had changed in code. The fact is, that Microsoft changed the memory sub-system in Vista to start pre-loading heavily used applications into memory (and naturally it tended to include their products even before first use), so reported use (nearly 100% memory utilization) wasn't realistic and we had to adjust.

 

The same concept should apply to the GPU as well. Hoard a texture in System RAM if it puts it there just in case, because it's faster to load than from disk. So, just because it is sitting in System RAM does not equate to it's requirement for use ever again in the current session.

 

My point is that speculating 1.X of overall GPU related memory use as being safe, without taking into consideration it's actual need, is misleading. I can only see a case for determining the frequency of swapping as being the only real measure of texture use to graphical anomalies. Consequently, a high frequency of swapping should line up with over-use of higher resolution textures, which would mean anything beyond 1.0 is immediately noticeable.

All well said, but it still does not explain the reality of points made previously. Please speculate on the relative change in dynamic and dedicated VRAM in the performance charts of the DDSopt guide. As I implied, I think that these should be taken as conservative "just in case" numbers, hence a soft threshold, hence a safe bet that we can exceed the reported VRAM threshold by some amount, and 1.1x seems perfectly reasonable, given the data