Here's my experience with Skyrim and memory usage:
Bought a new graphics card with lots of VRAM (3GB).
Already had a rig with LOTS of RAM (24GB).
Figured i'd install all these hi-res textures that i couldn't use before.
By the way, a symptom of running out of VRAM is, IMO, this: FPS is OK while you don't move, but it immediately drops when you turn around (i.e. the card can't keep the textures the stuff in front of you and behind you in memory simultaneously).
Anyway, i've installed all available hi-res textures for everything in STEP.
The game ran fine - but kept terminating itself while i was strolling around the world (note that "terminating itself" is the correct term to use here; "CTD" ("Crash To Desktop") has "crash" in it, which is the situation when the program throws an exception it is incapable of handling, and the OS steps in and kills it (you get a crash dialog from Windows); in this case the exception (if it was an exception; could be anything) WAS handled by Skyrim - handled by immediate process termination (another reason why i think that TES5 developers are not very competent)).
Watching TESV in ProcessHacker revealed that it always ran up to 3.2GB of Private Memory at the time of the crash. That hinted at memory problems. I ran Ordinator to optimize and scale down 4k textures into 2k, and that improved things a bit. Later on i ran it again, and scaled down to 1k that time. This reduced memory usage to 2.3-2.6GB instead of near-constant 3.2GB, and practically removed all self-termination issues.
The moral of this story: don't install hi-res textures over 1k (there are some packs that contain a mix of 1k and 2k, where 2k is only used for _some_ textures that really need the resolution - these are probably ok).
Obviously, there are other things than textures in memory, but textures, i guess, do take up lots of space. Case in point: a float takes 4 bytes, a vertex consists of 3 floats (4*3=12 bytes). A model consists of, say, 1000 triangular polygons. I'm too lazy to make a correct estimation of the number of vertices that would be needed for 1000 polygons (it depends on the shape of the model, obviously; less than the number of polygons - that is the only thing you can be certain of). Let's say 900 vertices. One polygon consists of 3 integers (vertex indices), an integer takes up 4 bytes, one polygon takes up 3*4=12 bytes. 900 vertices * 12 bytes-per-vertex = 10800 bytes; 1000 polygons * 12 bytes-per-polygon = 12000. 12000+10800=22800 bytes. That is, slightly more than 22 kilobytes. Compare to textures, which never go lower than 100 kilobytes (small ones, with good compression; uncompressed textures are measured in megabytes - 512*512*4-bytes-per-pixel (uncompressed 512x512 rgba) = 1 megabyte).
An open question is why TES5 keeps a copy of each texture in RAM (at least that is the impression i'm getting), when they should go into VRAM of the graphics card.
Hopefully, TES6 will have a 64-bit version that doesn't run out of 32-bit address space...