How to create grass textures for ENB Complex Grass

[MarcDwonn]

I managed to get NifSkope showing the textures, and here everything seems OK. I double-checked the texture source as well. It's all very confusing...

[z929669]

Do you know the exact meshes involved for each and which grasses each uses? How many different grass/mesh combinations have this problem?

Have you tested Skurkbro's is using those same meshes? Compare each impacted grass texture with Skurkbro's and that each one from that mod corresponds to your modified versions in terms of resolution/dimensions/atlas.

The axisgrass seems to be rendering only the normals, but I can't tell if that's the case with the other grasses affected.

Test if replacing only the affected grass textures in your mod with Skurkbro's versions fixes the problem, with no other changes at all.

If they are using the same meshes, then the issue is the textures, and I would guess it has something to do with NiAlphaProperty on the meshes and how that plays with the alpha test threshold of the textures. If resolution is exactly the same for diffuse/normal/atlas for yours and Skurkbro's, then we can't blame UV mapping.

Grass meshes are odd in that they almost never render the texture in NifSkope in my XP. How did you get the textures to render in NifSkope? Did you alter the NiAlphaProperty value in the mesh?

[MarcDwonn]

Quote

How many different grass/mesh combinations have this problem?

2 of 24. Exact meshes are known.

Quote

Test if replacing only the affected grass textures in your mod with Skurkbro's versions fixes the problem, with no other changes at all.

If i remove those 2 textures and use Skurk's instead, i have no issues:

Here's a closeup that demonstrates how the game renders both Diffuse and Normal Maps as Color texture, in my two problematic DDS files (same grass above and below; the transparency is caused by the black parts of the specular map (which the game things is an alpha map). Before you ask, yes, i placed 8px square 127-grey triggers into the specular maps, like in the guidelines. -

 

With NifSkope i was never "lucky" to get it to render the textures, if started inside MO2. So i start it standalone and as a texture resource i use the direct texture path to my mod in MO2's mod folder ("tundra test").

I'm gonna carefully compare my "bad" textures with Skurk's, again, and get back to you. BTW, what i did NOT test yet, is placing a trigger "pixel" onto the Normal Map as well. JohnSkyrim's guide said that it's not needed, if using a proper normal map, but what does that even mean? :shrug:

[MarcDwonn]

Interesting discovery: After comparing my "bad" texture with Skurkbro's, sure, the specular maps are completely different, but the trigger area in his textures is RGB 15,15,15 - which is way below the guidelines.

I took his Specular map into my "bad" texture, and the texture was fixed in-game! Then i just set a trigger area of RGB 15,15,15 and the issue was gone as well. Boris on the ENB discord told me that my gamma settings in Photoshop might be wrong and it's messing with the trigger areas. I'm gonna try disabling color management for the textures i open.

UPDATE: Issue found and fixed. I worked in sRGB in Photoshop, and it messes with the gamma. Working in "Untagged RGB" leaves the trigger areas at the correct 127,127,127 values and no more blue sheets in game. I'm ready to do all the grass now.

With sun at zenit (ignore the GrassLOD):

Edited July 21, 2023 by MarcDwonn

[z929669]

1 hour ago, MarcDwonn said:

I'm gonna carefully compare my "bad" textures with Skurk's, again, and get back to you. BTW, what i did NOT test yet, is placing a trigger "pixel" onto the Normal Map as well. JohnSkyrim's guide said that it's not needed, if using a proper normal map, but what does that even mean? :shrug:

That pixel trigger area is only to ensure that ENB can identify it as a normal map in case the normal map has some unexpected noise in that 'testing' part of the normal. I think 127 gray is the baseline of a 'blank' normal map. Lowering that value to 120/120/120 also works, but I believe it basically reduces the 'weight' of the ENB complex grass effect like SSS

[MarcDwonn]

Well, turns out that if the trigger area is above 127 grey, you get those funky blue sheets like i did in my game. It seems that the game doesn't recognize the DDS as a texture atlas anymore and uses the whole height as one single texture. It's beyond my technical understanding...

[z929669]

42 minutes ago, MarcDwonn said:

Interesting discovery: After comparing my "bad" texture with Skurkbro's, sure, the specular maps are completely different, but the trigger area in his textures is RGB 15,15,15 - which is way below the guidelines.

I took his Specular map into my "bad" texture, and the texture was fixed in-game! Then i just set a trigger area of RGB 15,15,15 and the issue was gone as well. Boris on the ENB discord told me that my gamma settings in Photoshop might be wrong and it's messing with the trigger areas. I'm gonna try disabling color management for the textures i open.

UPDATE: Issue found and fixed. I worked in sRGB in Photoshop, and it messes with the gamma. Working in "Untagged RGB" leaves the trigger areas at the correct 127,127,127 values and no more blue sheets in game. I'm ready to do all the grass now.

With sun at zenit (ignore the GrassLOD):

Looks good!

So I'm not clear on Ps "messing with the trigger area" ... the neutral gray (15/15/15 or 80/80/80 or 127/127/127) would only change if the trigger area was created and the entire specular map was changed. I never had such an issue in Ps, and I was using Ps RGB default RGB (not sure what it was). I don't think it matters as long as the trigger-area color modification is basically the last step in modifying the specular. Then simply copying it to the alpha of the normal just 'works' ... so maybe your workflow for these two textures was a bit different than the rest (maybe the first two normals you created?)?

5 minutes ago, MarcDwonn said:

Well, turns out that if the trigger area is above 127 grey, you get those funky blue sheets like i did in my game. It seems that the game doesn't recognize the DDS as a texture atlas anymore and uses the whole height as one single texture. It's beyond my technical understanding...

Yes, neutral gray < 127 reduces the ENB-CG effect proportionately, and values > 127 are invalid, so that makes sense.

YTou finished this pretty quickly, so kudos on that. It took me waaay more time

[MarcDwonn]

I was quick because i had your help, it saved me many hours of trial and error. Super appreciative about that! Plus i have experience with batch processing piles of stuff (been using Photoshop for over 20 years now, so i know my way around).

But this whole texture stuff is a real learning experience. And issues like above can completely stop me in my tracks, because then i don't know what to look for.

Color management is very complex - i only understand it enough to set it up in my software and my files. But under the hood i seems to change the gamma values of pixels: let's say the color managed monitor displays 127, but in the file the value is saved as 129. I always though it's the other way around - that only the display of the pixels is changed for color accurate view, not the actual values in the file, but i'm not sure anymore.

Funny detail: To automate the creation of the trigger area, i couldn't just make a selection in the lower left corner and fill it with 127 grey, because the textures are different size and the positioning is absolute. I made a Temp layer, made a 8px selection in the middle, filled it with color, auto-aligned it to the lower left corner, then CTRL+LM selected the 8px area, switched to the Alpha channel, filled the selection with 127,127,127, and deleted the Temp layer. Maybe it can be done more efficiently, but this was the "quick and dirty" way. Oh, that reminds me - i've still not automated the padding of the Alpha map. Got work to do...

Edited July 21, 2023 by MarcDwonn

[z929669]

57 minutes ago, MarcDwonn said:

I was quick because i had your help, it saved me many hours of trial and error. Super appreciative about that! Plus i have experience with batch processing piles of stuff (been using Photoshop for over 20 years now, so i know my way around).

But this whole texture stuff is a real learning experience. And issues like above can completely stop me in my tracks, because then i don't know what to look for.

Color management is very complex - i only understand it enough to set it up in my software and my files. But under the hood i seems to change the gamma values of pixels: let's say the color managed monitor displays 127, but in the file the value is saved as 129. I always though it's the other way around - that only the display of the pixels is changed for color accurate view, not the actual values in the file, but i'm not sure anymore.

Funny detail: To automate the creation of the trigger area, i couldn't just make a selection in the lower left corner and fill it with 127 grey, because the textures are different size and the positioning is absolute. I made a Temp layer, made a 8px selection in the middle, filled it with color, auto-aligned it to the lower left corner, then CTRL+LM selected the 8px area, switched to the Alpha channel, filled the selection with 127,127,127, and deleted the Temp layer. Maybe it can be done more efficiently, but this was the "quick and dirty" way. Oh, that reminds me - i've still not automated the padding of the Alpha map. Got work to do...

Yes, this is how it works, because the alternative would completely break the usefulness of the app. A simple test would be to create a small 127/127/127 default-RGB image and save it. Then analyze the values using some other application (e.g., open in a browser and use the dev tools to sample color, or open in Paint.net). Then you could test what modifications to that file might mess with this. I have NEVER had my hex color value mysteriously changed by any app unless I modified something else in the image that included the pixels I didn't intend to change. Now, the impacts of downstream apps like the game or the mesh on the rendered values is another matter, but I think everyone agrees that their digital specifications are expected to be maintained with something as basic as color, regardless of the rendered result on the monitor or in-game.

The target pixel resolution does depend on the total resolution, but I recall it not being a specific size or proportionate-size. Same with the top padding on the normal. I just made both big enough according to my eye. All of my assets were either 1K or 2K with like two 4K stragglers I did manually. This is my Ps macro for the trigger pixel area, so it looks like I use absolute size and position using a fill layer, depending on resolution of the image. Then saved as PNG downstream after creating the top padding in a similar manner, so all was merged on save. Building the atlas was a separate action later in the process:

That's just the way I did it, and it's likely not optimal either.

[MarcDwonn]

UPDATE:

I reworked three different grass mods: Folkvangr + Origins Of Forest for my current modlist, and then Cathedral 3D Pine Grass - Full 3D Coverage (in case i ever want to go back to QW's Patch 2). But, i had a couple of issues again, mostly due to shaky documentation. :sigh:

The ComplexGrass Guide about trigger area in Normal Map: "If your normal map is correct, you do not need to worry about this area." This is wrong, and i wasted quite a bit of time because of it. Even with proper trigger area 32x32px @ 127 grey in Specular Map, which i thought is taking care of my previous issue, i got many blue sheets on different grasses:

When i took the extra time and placed 127,127,255 patches on every normal map, the issue was completely gone.

Then, i probably misunderstood the CG guide saying that 4px Alpha padding is minimum but should work. This was the result with 10px padding (IIRC):

As a solution, i had to extend the black padding quite a bit into the Specular Map to get rid of it on some grasses.

Edited July 24, 2023 by MarcDwonn

[MarcDwonn]

Finally i tweaked the GrassLOD and the following is the result.

Mornings and afternoons look great:

But at noon i still have quite dark GrassLOD:

 

Is this the best that can be done with the current tools, or could i improve that some more?

 

My settings for TexGen are default (STEP v2.2), and for Dyndolod i'm using these:

ComplexGrassBillboard=5

ComplexGrassBacklightMask=35

; complex grass LOD brightness multipliers - applied to grass LOD billboards created from "complex" grass with normal map textures
ComplexGrassBrightnessTopR=0.750
ComplexGrassBrightnessTopG=0.750
ComplexGrassBrightnessTopB=0.750
; make bottom darker to fake shadowing
ComplexGrassBrightnessBottomR=0.600
ComplexGrassBrightnessBottomG=0.600
ComplexGrassBrightnessBottomB=0.600

 

I'm aware that this could be something that NAT3 makes worse, in comparison to, let's say, Cathedral Weathers. Honestly, the current results are good enough for me, but i'm curious if people have gotten better results, still.

Edited July 24, 2023 by MarcDwonn

[z929669]

1 hour ago, MarcDwonn said:

UPDATE:

I reworked three different grass mods: Folkvangr + Origins Of Forest for my current modlist, and then Cathedral 3D Pine Grass - Full 3D Coverage (in case i ever want to go back to QW's Patch 2). But, i had a couple of issues again, mostly due to shaky documentation. :sigh:

The ComplexGrass Guide about trigger area in Normal Map: "If your normal map is correct, you do not need to worry about this area." This is wrong, and i wasted quite a bit of time because of it. Even with proper trigger area 32x32px @ 127 grey in Specular Map, which i thought is taking care of my previous issue, i got many blue sheets on different grasses:

When i took the extra time and placed 127,127,255 patches on every normal map, the issue was completely gone.

Then, i probably misunderstood the CG guide saying that 4px Alpha padding is minimum but should work. This was the result with 10px padding (IIRC):

As a solution, i had to extend the black padding quite a bit into the Specular Map to get rid of it on some grasses.

My pixel trigger area and top padding was significantly bigger than the doc suggested, I think. My trigger is 32x32 px and all the way in the lower left corner @ 127/127/127. My top padding is 24 px ... just checked, and these absolute sizes appear to be fine for both 1K and 2K textures.

1 hour ago, MarcDwonn said:

Finally i tweaked the GrassLOD and the following is the result.

Mornings and afternoons look great:

But at noon i still have quite dark GrassLOD:

 

Is this the best that can be done with the current tools, or could i improve that some more?

 

My settings for TexGen are default (STEP v2.2), and for Dyndolod i'm using these:

ComplexGrassBillboard=5

ComplexGrassBacklightMask=35

; complex grass LOD brightness multipliers - applied to grass LOD billboards created from "complex" grass with normal map textures
ComplexGrassBrightnessTopR=0.750
ComplexGrassBrightnessTopG=0.750
ComplexGrassBrightnessTopB=0.750
; make bottom darker to fake shadowing
ComplexGrassBrightnessBottomR=0.600
ComplexGrassBrightnessBottomG=0.600
ComplexGrassBrightnessBottomB=0.600

 

I'm aware that this could be something that NAT3 makes worse, in comparison to, let's say, Cathedral Weathers. Honestly, the current results are good enough for me, but i'm curious if people have gotten better results, still.

I'm not sure if you are running 1.5.97 with NGIO and extending grass or if you are using 1.6.xxx and LOD. For the latter, your LOD grass won't react to light in the same way as the loaded grass, and this dark band looks like that.

I set all my T/B DynDOLOD brightness at 0.500 , since I want to keep the native colors in my LOD grass and use TexGen to control brightness. This is for Cathedral under Step:

 

ComplexGrassBillboard=5
ComplexGrassBacklightMask=25     ;try 0/50/100 to see what works best for your setup

... be sure to check under clear weather (TU is the least forgiving so best for testing. Some cloudy weathers are also unforgiving) and at dusk/dawn as well as noon. Unfortunately, you need to rund TexGen/DynDOLOD for each variation. I suggest doing so three times and placing the outputs in 6 different mods to test all at once.

There's also the ENB-CG settings to consider. This won't impact LOD grass. This is how our ENB is set up (ENB gurus' input was provided for these values):

[COMPLEXGRASS]
IgnoreWeatherSystem=false
IgnoreWeatherSystemInterior=true

ApplyToAllGrass=true
ApplyToBasicGrass=true
EnableSpecular=true
EnableSubSurfaceScattering=true
EnablePointLights=true
EnableInterior=false

SpecularMultiplierDawn=1.0
SpecularMultiplierSunrise=1.0
SpecularMultiplierDay=1.0
SpecularMultiplierSunset=1.0
SpecularMultiplierDusk=1.0
SpecularMultiplierNight=1.0
SpecularMultiplierInteriorDay=1.0
SpecularMultiplierInteriorNight=1.0
SubSurfaceScatteringAmountDawn=0.2
SubSurfaceScatteringAmountSunrise=0.4
SubSurfaceScatteringAmountDay=0.5
SubSurfaceScatteringAmountSunset=0.4
SubSurfaceScatteringAmountDusk=0.2
SubSurfaceScatteringAmountNight=0.2
SubSurfaceScatteringAmountInteriorDay=0.2
SubSurfaceScatteringAmountInteriorNight=0.2
BasicGrassSSSAmountDawn=0.2
BasicGrassSSSAmountSunrise=0.2
BasicGrassSSSAmountDay=0.2
BasicGrassSSSAmountSunset=0.2
BasicGrassSSSAmountDusk=0.2
BasicGrassSSSAmountNight=0.2
BasicGrassSSSAmountInteriorDay=0.2
BasicGrassSSSAmountInteriorNight=0.2
BasicGrassFakeLightDawn=0.0
BasicGrassFakeLightSunrise=0.0
BasicGrassFakeLightDay=0.0
BasicGrassFakeLightSunset=0.0
BasicGrassFakeLightDusk=0.0
BasicGrassFakeLightNight=0.0
BasicGrassFakeLightInteriorDay=0.0
BasicGrassFakeLightInteriorNight=0.0
FakeLightDawn=0.1
FakeLightSunrise=0.2
FakeLightDay=0.1
FakeLightSunset=0.2
FakeLightDusk=0.1
FakeLightNight=0.0
FakeLightInteriorDay=0.0
FakeLightInteriorNight=0.0
SpecularPowerMultiplierDawn=1.0
SpecularPowerMultiplierSunrise=1.0
SpecularPowerMultiplierDay=1.0
SpecularPowerMultiplierSunset=1.0
SpecularPowerMultiplierDusk=1.0
SpecularPowerMultiplierNight=1.0
SpecularPowerMultiplierInteriorDay=1.0
SpecularPowerMultiplierInteriorNight=1.0

 

 

 

[MarcDwonn]

13 hours ago, z929669 said:

I'm not sure if you are running 1.5.97 with NGIO and extending grass or if you are using 1.6.xxx and LOD. For the latter, your LOD grass won't react to light in the same way as the loaded grass, and this dark band looks like that.

Ah, that would explain it. Yes, i'm using Anniversary Edition with the latest EXE.

A year ago with the Elysium modlist i had the best success without using CG, and the far grass was quite consistent. Now with the Nordic Souls modlist, without using CG the far grass was very inconsistent, and even changed color completely at different TOD (now i understand why). That's the reason why i switched to CG.

I'm not gonna bother tweaking further - it seems that the v1.6.x solution is inferior and will not get much better. I reached my goal for the grass - to be immersive enough for my upcoming playthrough.

Thanks again for all the help, Z!

PS: If i ever invest the effort to downgrade to v1.5.97 to be able to use NGIO, i'll post my results here.

[z929669]

1 hour ago, MarcDwonn said:

Ah, that would explain it. Yes, i'm using Anniversary Edition with the latest EXE.

A year ago with the Elysium modlist i had the best success without using CG, and the far grass was quite consistent. Now with the Nordic Souls modlist, without using CG the far grass was very inconsistent, and even changed color completely at different TOD (now i understand why). That's the reason why i switched to CG.

I'm not gonna bother tweaking further - it seems that the v1.6.x solution is inferior and will not get much better. I reached my goal for the grass - to be immersive enough for my upcoming playthrough.

Thanks again for all the help, Z!

PS: If i ever invest the effort to downgrade to v1.5.97 to be able to use NGIO, i'll post my results here.

You should still be able to get a optimal near/far match with or without CG at all ToD under 1.6.xxx (DynDOLOD grass mode 1) by optimizing the TexGen brightness and the DynDOLOD backlighting settings, but it does require time and testing, so I understand it's likely overkill for your playthrough. After all, it's really only the tundra that grass can be clearly seen at distance and the noticeable loaded/LOD band becomes more obvious at certain ToD.

Those settings I posted are optimized for Cathedral Landscapes, and it did require a lot of LOD regen and testing to find them.