Jump to content
  • 0

Anti-Aliasing/Anisotropic Filtering not working properly

torminater

Asked by torminater,

 Share

Question

torminater Emperor

torminater

Posted
Posted

Hey guys,

 

today I found out that my texture shimmering is absolutely related to Anti-Aliasing and/or Anisotropic Filtering. Setting the "Allow negative LOD bias" to "Clamp" makes the structures appear halfway decent again, BUT it still is there.

So I started yet another testing round using only Skyrim assets and Vanos Hybrid HD-DLC-Mod.

 

 

That's what I did:

1. Disabled Bethesda Logo on startup.

2. Enabled logging and tracing in skyrim.ini for papyrus.

3.Opened up Skyrim Game Launcher OR Skyrim.ini

4. Applied various settings. Details below

5. Enabled "Boost"-mode in IObits Game Booster.

6.Started Skyrim Performance Monitor.

7. Ran 100% clean Vanilla Skyrim 1.6.89 with TESV.exe (sry not SKSE!!!).

6. Loaded new game (started today for testing purposes only) from exiting Helgen Keep dungeon.

7. God mode enabled.

8. Continiously sprinted to Whiterun only travelling by road ignoring all combat except a part of the giant fight before Whiterun. Ended testing whilst looking down from Dragonsreach and testing VRAM-Stuttering there.

9. Exited Skyrim.

10. Screenshot of Skyrim Performance Monitor.

 

First test:

Skyrim Launcher Options:

Lowest, means Low Preset applied and manually set everything except the resolution to the lowest possible setting, kept object detail fade as it gets preset.

Ini Options:

no changes

Data:

Posted Image

Comments: I actually managed to crash once to desktop, where TESV.exe was apparently not responding anymore.

 

 

Second test:

Skyrim Launcher Options:

Low Preset

Ini Options:

no changes

Data:

Posted Image

Comments: I actually managed to crash once to desktop, where TESV.exe was apparently not responding anymore.

 

 

Third test:

Skyrim Launcher Options:

Medium Preset

Ini Options:

no changes

Data:

Posted Image

Comments: Nothing.

 

 

 

Fourth test:

Skyrim Launcher Options:

High Preset

Ini Options:

no changes

Data:

Posted Image

Comments: Nothing.

 

 

 

Fifth test:

Skyrim Launcher Options:

Ultra Preset

Ini Options:

no changes

Data:

Posted Image

Comments: Aliasing on stairs, lit up items with shadows cast on them (plus shadow striping/strobing and all that crap) and items on loading screens. The game felt a little bit laggy, but bearly noticeable only. No lag/stutter on the Dragonsreach reference spot.

 

 

 

Sixth test:

Skyrim Launcher Options:

Ultra Preset plus original High-Res-DLC from Bethesda

Ini Options:

no changes

Data:

Posted Image

Comments: Aliasing on stairs, lit up items with shadows cast on them (plus shadow striping/strobing and all that crap) and items on loading screens. The game felt a little bit laggy, but bearly noticeable only. No lag/stutter on the Dragonsreach reference spot.

 

 

 

Seventh test:

Skyrim Launcher Options:

Ultra plus original High-Res-DLC from Bethesda plus Vano89s optimized Hybrid HD-DLC-version

Ini Options:

no changes

Data:

Posted Image

Comments: Aliasing on stairs, lit up items with shadows cast on them (plus shadow striping/strobing and all that crap) and items on loading screens.  The game felt a little bit laggy, but bearly noticeable only. No lag/stutter on the Dragonsreach reference spot.

 

 

 

Eighth test:

Skyrim Launcher Options:

Ultra plus original High-Res-DLC from Bethesda plus Vano89s optimized Hybrid HD-DLC-version

Ini Options:

uGridsToLoad=7

uExterior Cell Buffer=64

Data:

Posted Image

Comments: Aliasing on stairs, lit up items with shadows cast on them (plus shadow striping/strobing and all that crap) and items on loading screens.  The game felt a little bit laggy, but bearly noticeable only. Not really lagging/stuttering more than with normal uGrids-settings... No lag/stutter on the Dragonsreach reference spot.

 

 

  Reveal hidden contents

 

 

Nineth test:

Skyrim Launcher Options:

Ultra plus original High-Res-DLC from Bethesda plus Vano89s optimized Hybrid HD-DLC-version

Ini Options:

uGridsToLoad=9

uExterior Cell Buffer=100

Data:

Posted Image

Comments: Aliasing on stairs , lit up items with shadows cast on them (plus shadow striping/strobing and all that crap)and items on loading screens.  The game felt a little bit laggy, but bearly noticeable only. Maybe lagging/stuttering more than with normal uGrids-settings...

 

 

  Reveal hidden contents

 

Tenth test:

Skyrim Launcher Options:

Ultra plus original High-Res-DLC from Bethesda plus Vano89s optimized Hybrid HD-DLC-version

Ini Options:

uGridsToLoad=11

uExterior Cell Buffer=144

Data:

Posted Image

Comments: Aliasing on stairs, lit up items with shadows cast on them (plus shadow striping/strobing and all that crap) and items on loading screens. More lagging/stuttering than with normal uGrids-settings... Game freezed as I closed in on the Whiterun periphery.

 

  Reveal hidden contents

 

  • Answers 109
  • Created
  • Last Reply

Top Posters For This Question

Popular Days

Top Posters For This Question

Popular Days

109 answers to this question

Recommended Posts

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted

this is odd, now I'm getting the same crashes on a new game that you got on the first post right before riverwood.

 

google'd for "DialogueRiverwoodIntroScene (000954D9): attempting to start event scoped quest outside of story manager." and your post came up lol

  • 0
torminater Emperor

torminater

Posted
  • Author
Posted

then it's probably a vanilla bug... ;-)

well, actually my rig crashed on the lvl predator script just above that:

 

  Reveal hidden contents

 

 

the dialogue script you refer to and mentioned in the spoiler only came up after installing USKP 1.05 c beta

  • 0
Neovalen Akatosh

Neovalen

Posted
Posted

Maybe not the right place to ask this, but since we're discussing various Nvidia Inspector settings... I've set up 4x4SSAA/4xTrSGSSAA etc as noted in the first few posts of this thread.

 

However, my question for everyone is... Nvidia SSAO vs ENB SSAO vs Both? I'm confused as to which I should use.

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted
  Quote

Maybe not the right place to ask this, but since we're discussing various Nvidia Inspector settings... I've set up 4x4SSAA/4xTrSGSSAA etc as noted in the first few posts of this thread.

 

However, my question for everyone is... Nvidia SSAO vs ENB SSAO vs Both? I'm confused as to which I should use.

I too am confused, but I saw no difference between off and on when using ENB. I think ENB uses it's own type of it, since it's definitely a lot better than regular. and it seems to switch off the regular when you enable it (I think, anyways) so I leave it off in nvidia inspector. if you find different results, let me know :D

 

Edit: also, don't use 4x4 supersample. it doesn't work, and even if it did your framerate would be like a slideshow. especially with your res, it would be rendering at 23040x4320. the game's setting overrides nvidia inspector anyways, but I set 4x MSAA override with 4x in-game just for good measure. (I saw some kind of fps boost when doing so, but could have been within margin of error)

 

Edit2: if your cards can handle it (which they should) I highly recommend 4x SGSSAA instead of 4x TrSSAA, it produces a much better image quality - gets rid of most of the shimmering from lizard skins, doors, leather helmets, standing stones, and other specular objects, etc etc

 

Edit3: and I think the jury's still out on matching MSAA with TrSSAA/SGSSAA but I do it for good measure. (like 4x msaa + 4x trssaa/sgssaa, and 8x msaa + 8x trssaa/sgssaa). someone else had mentioned it only matters with supersample AA (2x2, 3x3, 4x4) and not MSAA but I'm not entirely sure. I don't usually like to use more than 4x MSAA anyways since 8x and above are a much bigger performance hit.

  • 0
Neovalen Akatosh

Neovalen

Posted
Posted
  Quote
  Quote

Maybe not the right place to ask this, but since we're discussing various Nvidia Inspector settings... I've set up 4x4SSAA/4xTrSGSSAA etc as noted in the first few posts of this thread.

 

However, my question for everyone is... Nvidia SSAO vs ENB SSAO vs Both? I'm confused as to which I should use.

I too am confused, but I saw no difference between off and on when using ENB. I think ENB uses it's own type of it, since it's definitely a lot better than regular. and it seems to switch off the regular when you enable it (I think, anyways) so I leave it off in nvidia inspector. if you find different results, let me know :D

 

Edit: also, don't use 4x4 supersample. it doesn't work, and even if it did your framerate would be like a slideshow. especially with your res, it would be rendering at 23040x4320. the game's setting overrides nvidia inspector anyways, but I set 4x MSAA override with 4x in-game just for good measure. (I saw some kind of fps boost when doing so, but could have been within margin of error)

 

Edit2: if your cards can handle it (which they should) I highly recommend 4x SGSSAA instead of 4x TrSSAA, it produces a much better image quality - gets rid of most of the shimmering from lizard skins, doors, leather helmets, standing stones, and other specular objects, etc etc

 

Edit3: and I think the jury's still out on matching MSAA with TrSSAA/SGSSAA but I do it for good measure. (like 4x msaa + 4x trssaa/sgssaa, and 8x msaa + 8x trssaa/sgssaa). someone else had mentioned it only matters with supersample AA (2x2, 3x3, 4x4) and not MSAA but I'm not entirely sure. I don't usually like to use more than 4x MSAA anyways since 8x and above are a much bigger performance hit.

Alright... 4xMSAA doesn't work with Stereoscopic 3d enabled. I get a big red message from Nvidia saying it's been downgraded to 2xAA.... without 3d enabled it works fine.

Of course I don't get HDR then due to SLI.... gotta love it. Also noted that SSAO from Nvidia DOES NOT WORK since patch 1.5... Beth screwed something up when they implemented grass shadows.

 

Is it bad I'm actually HOPING for a patch from Bethesda? All of this is reminding me why I stopped playing Oblivion after too long even though I had fun with the game itself.

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted

ENB's SSAO is pretty nice, there's no need for regular SSAO really, though it's good to know that it's broken. there's very vanilla-ish ENB's with minimal performance hit (2-3) like Enhanced Vanilla ENB. (just disable depth of field if you don't like that). it even works with RCRN as a dll proxy

 

personally I'm using Vibrant ENB with the vibrancy turned off, combined with rcrnshaders.esp for darker nights

  • 0
jmad75 Citizen

jmad75

Posted
Posted

Much of this discussion is lost on me. At this point, I don't know what my nvidia Inspector settings should be. GeForce GTX 560M is my card and I don't use SkyrimHD2K textures but I do have some shimmering issues.

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted
  jmad75 said:

Much of this discussion is lost on me. At this point I don't know what my nvidia Inspector settings should be. GeForce GTX 560M is my card and I don't use SkyrimHD2K textures but I do have some shimmering issues.

since your card is probably not fast enough for SGSSAA (sparse grid super sampling anti-aliasing), I would recommend using SMAA or FXAA, assuming you mean the shimmering on lizard skins, leather helmets, doors, standing stones, etc. SMAA does a bit better of a job than FXAA but not entirely a replacement for real transparency antialiasing.

 

there's a stand-alone SMAA injector here, but if you're using RCRN then that SMAA injector would replace RCRN's d3d9.dll file, so I would recommend uninstalling RCRN entirely else your contrast would be screwed up (or using an ENB that enhances contrast enough to replace RCRN's d3d9.dll). there's also a few ENB's that already include SMAA, like Vibrant ENB (though that ENB gave me a 10fps drop), but again these would replace RCRN. (though Vibrant ENB works very well with rcrnshaders.esp and textures)

 

as for FXAA, RCRN includes it by default (you can use rcrnconfigurator.exe to make sure it's on) and most ENB's include it. and then there's the in-game one. it will reduce the shimmering maybe halfway at the cost of extra blur, but not entirely get rid of it.

 

but if you really want to try SGSSAA, try 2x first. 4x will probably give you a massive drop in framerate.

 

Edit: you could also try the recommended 2x TrSSAA (Transparency Supersampling) to reduce tree leaf flickering, but it wont' do anything for specular highlights and it comes at a performance hit, though not as much as SGSSAA. I still recommend SMAA/FXAA on a laptop though, as the framerate will be much more playable

  • 0
Neovalen Akatosh

Neovalen

Posted
Posted

Alright, been a lot of speculation on this page on what does or does not work.... I'm just getting around now to re-installing Skyrim (moved drives and decided to make a clean run of it).

 

Things I have confirmed myself (Nvidia System):

1. HDR does not work for Skyrim in SLI unless Stereoscopic 3d is enabled. Stereoscopic 3d limits standard MSAA to 2x or lower.

2. Nvidia SSAO for Skyrim is broken as of patch 1.5+, so there is no need to enable it via Nvidia Inspector.

 

Things I am confused on:

1. What modes of AA actually work for Skyrim? Ansitropic Filtering = Clamp actually do anything? etc. Perhaps someone with some answers can post a definitive "best" setting?

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted

Regular AA:

I think it's safe to say that 2x2, 3x3, and 4x4 supersampling doesn't work (without a special comptibility flag). It also appears that forcing MSAA (like 4x) in nvidia inspector, while the in-game antialiasing is set to 0 or 1 doesn't work either. (though I only tried Override, not Enhance). So AA needs to be set in-game/INI. I set mine to 4x in-game and Override to 4x in nvidia inspector just for good measure. (I seemed to get a little more fps with override+4x too but unsure about that)

 

Transparency AA:

TrSSAA and SGSSAA both work, but I'm uncertain if 8x works as I got no performance hit over 4x. SGSSAA offers superior image quality since it reduces shimmering/flickering/aliasing on specular highlights like lizard skins, bumpy doors, and standing stones.

 

LOD Bias:

Clamp LOD bias shouldn't do anything over Allow really, since I don't think the game changes LOD bias natively (not 100% sure here), you'd only need Allow on if you intend to lower the LOD bias for sharpening. I just set Clamp for good measure since I don't want any sharpening (it can cause pixelation and further exacerbate aliasing). Plus, I believe the ENB's have a higher quality sharpening at the cost of a little performance (but don't quote me on that).

 

Anisotropic Filtering:

Definitely set to 16x in nvidia inspector or nvidia CP, it improves quality on water. Disable in-game/INI anisotropic for good measure and increased performance.

 

 

 

This is from my own testing, but I think torminater will agree here. Please correct me if I'm wrong on anything though.

  • 0
Neovalen Akatosh

Neovalen

Posted
Posted

According to the RCRN page, "mirror" values have to be set in the game itself for all the effects to work from the CP.

  Quote

After tuning the Driver Settings as listed in the Nvidia Tweak Guide, you have to do some things not reported in it, you have to set the right options for Skyrim.

In order to set them you have to start Skyrim via the Launcher and click on Options.

Here you have to set the “mirror” values of what you set on the driver panel, for example, if you set AF 16x and AA 8x forced by the Nvidia Control Panel, you will have to set the same values also into the in-game options, under the Antialiasing and Anisotropic Filter tabs.This thing is really important, and is necessary, because with the latest Skyrim’s patch only using the forced values without also having set the same values in the in-game options won’t work perperly.About the Nvidia FXAA, it’s really well done, so if you have a good rig we suggest to force it via the driver panel and use it along with the ones included in Skyrim (required, also if you force the Fxaa via panel) and RCRN.

I cannot confirm or deny on that, but their suggested settings for "Ultra" are as follows:
  Quote

NVIDIA Control Panel:

FXAA: ON

Texture Filtering: High Quality

Anisotropic Filter: AF 16x

Transparency Antialiasing (TRAA): 8x Supersample

Skyrim:

Antialiasing: AA 8x

Anisotropic Filter: AF 16x

Detail: Ultra

- Advanced: FXAA ON

RCRN:

FXAA: ON

Link: https://www.rcrncommunity.com/nvidia
  • 0
Neovalen Akatosh

Neovalen

Posted
Posted

After reading the notes from RCRN and doing some internet research I managed to force 4x4 Supersampling.

Unfortunately, Skyrim does NOT like it and has artifact issues (white shafts of light/flickering).

FPS was ~30 with no lighting mod @ 1080p.

 

Here was the settings to get it working:

IN GAME: 2xAA, disabled AF, FXAA enabled.

NVIDIA INSPECTOR settings attached.

Screenshot of artifacts also attached (sorry for the poor quality, it only allowed me to attach 500kb).

 

That being said, setting the Nvidia Inspector AA to 8x all the artifacts went away and I didn't see any jaggies ANYWHERE. Trying to use CSAA was also fruitless (game crashed / black screen).

I also didn't see any FXAA blurring as I think the latest Nvidia drivers took pretty much all of it out. (Using 304.79)

 

Also note: My VRAM usage is ~2500 even in just the opening sequence in Helgen. I strongly suggest not turning it all the way up without having a 4GB card.

  • 0
Neovalen Akatosh

Neovalen

Posted
Posted

Wanted to come back and note, disabling FXAA in both game and Nvidia Inspector had no visual difference than standard 8x MSAA forced in the Inspector. So I left it disabled from now on.

This way of using AA also bypassed Nvidia's check for >2xAA in Stereo3d.

  • 0
LeetMiniWheat Guard

LeetMiniWheat

Posted
Posted

Hmm, interesting about the Anisotropic filtering... wonder what the difference is. Anisotropic 16x with 1x set in-game has been working for me (I think), else I would have noticed blurry roads.

 

I still have my doubts about 4x4 supersampling, I just don't see how it could possibly be rendering a scene at your resolution TIMES FOUR. Was there a massive FPS hit after enabling it? In most games that it DID work in, I could barely even run 3x3 without a massive fps hit (even really old games). So, It could just be ignoring the setting... If it *is* working, you shouldn't need any TrSSAA/SGSSAA at all, or only 2x to get rid of the remaining minor jaggies on specular highlights.

 

I still think for 2x2, 3x3, or 4x4 supersampling it's gonna need one of those generic compatibility flags that causes artifacts and other issues in most games. Did you have to change the "Antialiasing - Behavior Flags"? because mine is set differently and I haven't touched it. But I doubt that would enable 2x2/3x3/4x4 usage, though in some games it "just works".

 

FXAA had major blurring last time I tried it. I believe the reason you didn't see a difference was because you had FXAA usage disallowed in nvidia inspector (but not totally sure here).

 

And regarding FPS, I'm trying to stay at 60fps/vsync all the time, 30 feels too sluggish for me. I'm strongly considering trading my 690 for two 4GB 670's though after seeing my VRAM usage. (when EVGA releases the 4GB FTW)

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to question listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...

Important Information

By using this site, you agree to our Guidelines, Privacy Policy, and Terms of Use.

  I accept