How to create grass textures for ENB Complex Grass

[MarcDwonn]

I used the current STEP guide (2.2.0) for TexGen and Dyndolod settings to setup my GrassLODs, and i'm happy with the results (a noticeable improvement in GrassLOD quality over v2.1.0). What i'm not happy with is my grass mod of choice - the quality of the complex grass textures is not good and i want to tweak those myself. I'm using QW's Grass Patch 2, which combines Folkvangr, Origins Of Forest and Cathedral 3D Grass in a very nice way in different regions. The CG Patches are from Skurkbro's compendium.

When the light comes from the side, everything looks quite good:

But when the sun is roughly at noon, the foreground grass gets too dark. The GrassLOD as well, btw, but that's not completely avoidable, AFAIK.

I tested this in sunny weather as well, where the issue is more severe, but this is the only good enough comparison i could find in my screenshots. Never mind that it's not quite noon in the second shot...

I remember reading that with better normal maps this can be minimized. We know that Skurk's patches don't have the best quality, so i want to tweak the textures in Photoshop myself.

Any suggestions what i could do to improve the behavior under different lighting angles? Just redo the normal maps with stronger settings? What about the specular maps? Do those need to be changed in any way, potentially adjusted down for less harshness? It would be nice to find good settings and then automate the changes / batch process everything in Photoshop.

PS: Just for reference - i'm using ComplexGrassBillboard=5 and ComplexGrassBacklightMask=35 in Dyndolod as well. My ENB is Rudy for NAT3.

[z929669]

See this sticky on Nexus and play with an 'alt2' grass billboard and ComplexGrassBacklightMask in DynDOLOD_SSE.ini.

[MarcDwonn]

Maybe you misunderstood me, maybe i wasn't clear enough. Currently i don't really care about the GrassLOD, i can tweak that later, but i find it good enough.

What i have an issue with is the darkness of the foreground grass with applied CG patches. It's the same without ENB, btw, so it's not ENB's fault. Worse - i tried making it brighter in ENB, but the difference was negligible.

I know that you guys patched Cathedral Grass for CG (for STEP), so i thought i could ask your expertise.

Edited July 17, 2023 by MarcDwonn

[z929669]

40 minutes ago, MarcDwonn said:

Maybe you misunderstood me, maybe i wasn't clear enough. Currently i don't really care about the GrassLOD, i can tweak that later, but i find it good enough.

What i have an issue with is the darkness of the foreground grass with applied CG patches. It's the same without ENB, btw, so it's not ENB's fault. Worse - i tried making it brighter in ENB, but the difference was negligible.

I know that you guys patched Cathedral Grass for CG (for STEP), so i thought i could ask your expertise.

Sorry, I should have read your post more thoroughly.

What you see in the 'loaded' grass is specifically governed by the normal maps on the atlas of each grass texture. You would not want to mess with the diffuse area of these textures (top half) so much as the normals.

What I found to work best --without going into a 3D application to develop the normal/specular maps based on the 3D models, which is far more advanced and time-consuming approch for beginners-- is to generate very noisy normals from the grayscale PNG versions of the original diffuse. This basically means that the tangent-space normals have a lot of surface 'diffusion' and are quite colorful as a result. Fine and medium details are most noisy, and large details less so. You can compare the normals I made in The CL-CG grass with those made for that same grass by others. Those I made are typically much noisier. This effectively mutes the reflectivity of the grasses a bit much like a glossy surface compares to a matte surface. Both reflect light, but glossy is much more reflective (which is the problem you are getting with low vs high sun angle).

When making the final normal, you want to use the specular map of that same grass as the alpha of the normal. This means that the lightest areas of the specular will allow more of the normal to be visible through the alpha transparency (black is opaque and white is fully transparent) ... so you don't want the specular to be too 'strong' or bright.

A tool like CrazyBump makes texture preparation a bit more convenient, but lots of other tools provide similar functionality. I use Photoshop to merge the final normals with specular alpha and to create the atlas with diffuse+normal.

[z929669]

Here's the progression for one particular CL grass.

Re-worked diffuse

Grayscale

normal

Specular

Final (with ENB pixel in alpha and alpha channel visible. Diffuse uses alpha cutout, and normal uses specular as alpha)

[z929669]

I should also note that testing with ENB enabled and CG disabled in the ENB will not be exactly the same result as testing without ENB.

Further, the "without ENB" testing should technically be done without the grass-atlas textures ... but I'm not positive on that. I don't recall if the atlas version behaves the same without ENB as the standard diffuse-only textures. Maybe it's good enough. It's definitely different in terms of LOD though, so I would ignore LODGen while you are working on them.

[MarcDwonn]

Thanks, Z! This is the detailed info i hoped to get here.

With "very noisy normals" do you mean adding extra noise, or just exaggerating detail when exporting for the "bump process", or does software like Crazybump have an extra Noise-Setting?

I'm gonna need a bit of time to absorb the knowledge and prepare. For example i've never worked with normal maps, and i'm lacking the tools. Besided, i'm still using Intel's TextureWorks (easy to use). Not looking forward to learn Nvidia's Tool, because it will freeze on me with anything else than default settings, but it is what it is...

Gonna post about my progress here. Sadly, for me there's no alternative to this grass mod, and i have to use CG, because the old style GrassLOD doesn't work anymore (or at least for me with my new fav weather - NAT3). I hope it doesn't prove too difficult. Me and Photoshop are good old friends, but assets for 3D application are quite a different neighborhood.

[z929669]

3 hours ago, MarcDwonn said:

Thanks, Z! This is the detailed info i hoped to get here.

With "very noisy normals" do you mean adding extra noise, or just exaggerating detail when exporting for the "bump process", or does software like Crazybump have an extra Noise-Setting?

I'm gonna need a bit of time to absorb the knowledge and prepare. For example i've never worked with normal maps, and i'm lacking the tools. Besided, i'm still using Intel's TextureWorks (easy to use). Not looking forward to learn Nvidia's Tool, because it will freeze on me with anything else than default settings, but it is what it is...

Gonna post about my progress here. Sadly, for me there's no alternative to this grass mod, and i have to use CG, because the old style GrassLOD doesn't work anymore (or at least for me with my new fav weather - NAT3). I hope it doesn't prove too difficult. Me and Photoshop are good old friends, but assets for 3D application are quite a different neighborhood.

It's been a while since I was into this one, so I forget some of what I did prescriptively when working on CL-CG ...

Load up the grayscale diffuse PNG.

First thing is to choose a 'shape' ... I think I used the right one, but it doesn't matter so much with plants/foliage. Just be consistent. It matters a lot more with shapes having an expected appearance like a ball or a bowl:

Looks like I saved my preset, and here's what it looks like I probably did for most of them (you can create both in the same session ... see tabs):

Mind you that I'm not a professional GFX artist. Just an amateur with a lot of XP

I found these settings after many hours of testing results in game, because I wasn't happy with just about any CG grass mod I tested. All of them had too much reflectivity that it just looked bad in game at most ToD, IMO.

The 'correct' way to do it is in the 3D application using standard procedures ... but that stuff doesn't apply to Skyrim and other Bethesda games, IMO. The engine is too old and limited to follow protocol.

 

Lastly, when saving out the final DDS, I recommend using NVIDIA Texture Tools Exporter (Ps plugin or standalone). Save as BC7 with 'normal' compression quality and alpha-test threshold of 127 (but this can vary a bit, depending on the textures):

Then there's the pixel and alpha trim at top of the normal to set as well, which takes a bit of doing to figure out. It's simple really but critical and can be tricky to figure out at first. Can't really help here so much, but Photoshop has lots of help with Google.

Download the CG specifications guide here. Lots of stuff from that is beyond what you are doing, but it gives you the CG specifications nicely.

Tip: record macro in Ps helps with > 3 grass textures when you are ready to build the final product. It's a lot more tedious work than it seems.

[MarcDwonn]

Sorry, didn't have time yet to reply. Just wanted to say that i highly appreciate your detailed help. A picture says more than thousand words, as they say. I can't think of anything more that i'd like to know ATM. I'll figure it out as i go.

The CG guide from JohnSkyrim i've read already, so i'm suffiently prepared, i'd say. Will post an update soon.

 

[MarcDwonn]

I'm studying your treatment of the above grass texture [fieldgrassTU (1)], and the diffuse is quite a mess in the original.

Orig:

Reworked:

(sorry for the desat images, my monitor is wide gamut and color management changes the colors of screenshots)

What i don't get is, how do you clean it up like that? It looks automated, like a software algo is taking the diffuse colors at the border of the alpha mask and interpolates them into the empty space around the mask. Is that a Photoshop filter i'm not familiar with? I've seen this "look" on other textures as well, but always thought that it's the result of the creator color-brushing over parts of the diffuse to bring in some variations (highlights, fake shadows etc.).

Edited July 20, 2023 by MarcDwonn

[z929669]

7 hours ago, MarcDwonn said:

I'm studying your treatment of the above grass texture [fieldgrassTU (1)], and the diffuse is quite a mess in the original.

Orig:

Reworked:

(sorry for the desat images, my monitor is wide gamut and color management changes the colors of screenshots)

What i don't get is, how do you clean it up like that? It looks automated, like a software algo is taking the diffuse colors at the border of the alpha mask and interpolates them into the empty space around the mask. Is that a Photoshop filter i'm not familiar with? I've seen this "look" on other textures as well, but always thought that it's the result of the creator color-brushing over parts of the diffuse to bring in some variations (highlights, fake shadows etc.).

I cleaned up all of the noise on the original alpha cutouts, many of which had lots of holes (white specs). So most of the problematic alphas were cleaned of specs, edges corrected where necessary, and saved with aliasing. EDIT: I wound up preferring the aliased over anti-aliased to eliminate some of the edge artifacts downstream in the process and would up using these as the final alpha, too. Ideally, I would have preferred to use very slight anti-aliasing, but the source was just too limited.

Example of obvious 'holes' that would show these specs on the mesh in final render in-game. There's a lot of small ones that can't be seen easily at screen-capture resolution in many of the source files (2K):

Before --> Cleaned

Using the corrected alphas, I cut out the diffuse to yield a transparent background, inverted the alpha mask, and used Flaming Pear / SolidifyC filter in Ps to yield a more 'bleed-proof' background to prevent artifacts at varying mipmap levels related to alpha test. Note that I also very-slightly darkened the resulting background, because grasses in this game look better this way, because higher mip levels (farther away) better match the first level (up close).

After I had the final diffuse, I saved out the diffuse grayscale as PNG (cutout only with transparent background). I probably changed the B/W curve a bit to yield slightly higher contrast. This was the basis for the normals and specular.

Flaming Pear (Free Plug-ins at bottom of page)

 

 

 

[MarcDwonn]

Great info! Never heard of Solidify, but it seems kind of a standard for this stuff. For the alpha a lazy Threshold works best. I tried to get as close as possible to your settings. This is the result:

 

That would cover the diffuse and alpha part of the exercise. Gonna get CrazyBump and see about Normal and Specular.

A plan for automating part of this stuff is already in the works.

Edited July 20, 2023 by MarcDwonn

[MarcDwonn]

CrazyBump works well, i was able to easily reproduce your settings after exporting a b/w PNG with transparency.

Comparing the end results with Skurkbro's fieldgrassTU (1).dds, it seems to me that the biggest difference is in the specular map. The normal maps are quite equal in "noisiness", the biggest differences are that Skurk has gone with the opposite shape and it looks a bit less flat (?).

I'm gonna isolate the tundra grasses for treatment, and run some tests in-game, very curious to see what will change...

[z929669]

1 hour ago, MarcDwonn said:

CrazyBump works well, i was able to easily reproduce your settings after exporting a b/w PNG with transparency.

Comparing the end results with Skurkbro's fieldgrassTU (1).dds, it seems to me that the biggest difference is in the specular map. The normal maps are quite equal in "noisiness", the biggest differences are that Skurk has gone with the opposite shape and it looks a bit less flat (?).

I'm gonna isolate the tundra grasses for treatment, and run some tests in-game, very curious to see what will change...

If Skurkbro's specular is overall brighter, the the diff will be in the normal alpha and likely will have more reflectivity.

As I mentioned previously, I don't think the shapes make much difference with respect to flat things like leaves. The main difference is in the 'direction' of the 'light' (shading) creating the 3D. As long as they are all consistent, it shouldn't matter ... it may not even matter that they are consistent.

[MarcDwonn]

Alright, since i switched to the combo Folkvangr + Origins Of Forest, i had to do the initial test with 24 textures (that's how much Folkvangr uses in the tundra near Whiterun :) )... But all worked well, those CrazyBump settings are excellent, the new textures are very well behaved, under all times of day. I already have my PS Actions in place, so i will process all 160 textures. (But damn, that CrazyBump really likes to crash...)

I have on issue though. Two of the reworked 24 textures show blue sheets. I redid them from scratch, but nothing changed. I compared to Skurkbro's and to the other textures that do work, but can't for the life of me find any differences.

Here's what it looks like in game (i had a trim problem as well in one texture in the screenshots, but it's already fixed):

 

 

Here's the third screenshot:

And here's what one of the NIFs look like in NifSkope. This is unfortunately beyond my expertise:

 

At first i thought those are missing textures, but no. It's just that the NIF seems to use the Normals part as Diffuse texture as well, and it calculates transparency from the Specular map (screenshot #3). I never worked with meshes, so if it's a NIF issue, i have no clue how it could be fixed.

Edited July 21, 2023 by MarcDwonn