I followed the discussion above for a while and with testing while playing I found a workaround which worked with reasonable results.
1. Use reduced texture sizes if the quality gap is not too bad (I would prefer to have better quality)
2. Use the Skyrim Performace Monitor https://skyrim.nexusmods.com/mods/6491
3. Use the console command "pcb" if the RAM usage is close to 3 GB
4. Use following settings in skyrim.ini
[General]
uExterior Cell Buffer=6
uInterior Cell Buffer=1
;clears the cell buffer every time you change the cell
bPreemptivelyUnloadCells=1
These settings reduce the usage of RAM and VRAM, but will result in increased disk IO, which might lead to stuttering in some game environments (I didn't get any stuttering).
The best effect I got with reduced texture sizes and the setting in skyrim.ini. The reduced texture sizes resulted in 2.5 to 3.0 GB, the ini-settings lead to 2.0 to 2.5 GB.
