Optimizing Texture Alpha Threshold

[Mousetick]

Recap of essential info:

• DynDOLOD 3a148 / Resources SE 3a45
• Vanilla tree meshes with Pine Branches Redone texture replacer
• Level0 at LOD Level 4
• DynDOLOD Resources SE's 3D tree LOD models for vanilla trees
• Edited to add UseMipMaps to crown nodes
• In the process of adjusting NiAlphaProperty Threshold to make LODs thinner

Now that the atlas textures are using the full texture mipmaps, I'm running into transparency/loss of details issues at smaller resolutions.

Look at the trees near the house in the center of the pictures below. Notice there is no difference between 112 and 127, and at 128 the LOD is very very close to the full model, which is fantastic.

LOD Alpha Threshold 112 > 127 > 128 > Full model

However notice the trees further in the distance to the right of the house have been severely "shaved" at Alpha threshold 128. It gets worse with smaller mipmaps, looking at the trees far away at the foot of the mountains:

It would seem that the full texture's mipmaps were not created with alpha-to-coverage, or they are, but with some kind of cutoff point for alpha around 127.

Based on the following shots, does it look to you like the mipmaps have alpha-to-coverage?

I see on Microsoft's Texconv page that the -keepcoverage option takes an "alpha test reference value". If the mipmaps of the full texture were generated with an "alpha test reference value" of 0.5, could this explain the observed behavior, i.e. no change between 112 and 127, and then "shaving" at 128?

What value does DynDOLOD use when it generates the atlas mipmaps (i.e. when not using UseMipMaps)?

Thanks in advance for your input.

[DoubleYou]

28 minutes ago, Mousetick said:

All thresholds of what? Same as what? Which transition? Please be specific.

I explained that I want to use NiAlphaProperty Threshold of 128. This is in order to preserve the alpha channel of the original branches texture's mipmaps when copied to the object LOD atlas, so that there are zero differences between the LOD mipmaps and the full model mipmaps.

Did you test 0.5 for the alpha coverage of the texture in Gimp? It should be basically identical to 0.51. This is the threshold I'm talking about. Gimp's alpha coverage is the most sensitive that I have found. Texconv at same threshold level is thinner IME.

[Mousetick]

1 hour ago, DoubleYou said:

Did you test 0.5 for the alpha coverage of the texture in Gimp? It should be basically identical to 0.51. This is the threshold I'm talking about. Gimp's alpha coverage is the most sensitive that I have found. Texconv at same threshold level is thinner IME.

No I didn't test 0.5 with GIMP. It's possible that it yields the same results as 0.51, I don't know. It seems each tool uses its own input value and formula to derive the alpha test value for the alpha-to-coverage filter. Z's Nvidia tool for example has an input value between 0 and 255, at least it's clear. Sheson found out that Texconv uses alpha-test-value=(input-value * 255) + 0.5, which rounds the fractional value to the nearest integer. We'd have to look at the GIMP source code to figure out what formula it uses.

Both alpha-to-coverage and LOD binary alpha use an alpha test value. Alpha-to-coverage doesn't test individual pixels, it's much more complicated. But we can still use a ballpark guesstimate rule-of-thumb. If we use an alpha test value a little greater than 128 - that is the alpha test value of LOD binary alpha - for the alpha-to-coverage filter, then there should be very little loss of detail (we're talking about single pixels at such resolutions as mipmaps) between full model mipmaps and LOD mipmaps when rendered on screen. They should also look very similarly opaque or transparent.

[z929669]

I know we continue to go down a slightly OT rabbit hole here, so I can move the relevant posts into a DynDOLOD Support question topic of it's own for posterity once we're finished ...

7 hours ago, sheson said:

The problem is the original source texture is being replaced by a full texture with mipmaps that do not have alpha-to -coverage. While the full model fades into nothingness until it switches to LOD, the LOD has a texture that doesn't. This can quickly be mitigated by adjusting the alpha threshold of the LOD model, so that the two texture roughly match when they switch, with is not a constant distance/mipmap or with more work by using the same texture unmodified texture for both. Which you achieved.

This makes a lot of sense to me now. Thanks

4 hours ago, sheson said:

DynDOLOD uses Texconv to convert compression formats only. The formula for scaled alpha coverage for a mipmap is alpha *= 1 + MipMapLevel * AlphaFactor.
MipMapLevel is 0 for the largest mipmap, so no change. DynDOLOD_SSE.INI sets AlphaFactor=0.25 because each mipmap is 1/4 of the resolution of the one before it.

I'm confused, since this is missing a variable/coefficient and has no perns indicating order of operations [ alpha * {blank} ] = (1 + (MipMapLevel) * AlphaFactor) ]

If MipMapLevel = 0 then either: (1 + 0) * AlphaFactor = AlphaFactor OR 1 + (0 * AlphaFactor) = 1 ... so I'm guessing it's the second order of operations and the first '*' is just a typo? Then: [ alpha = 1 + (MipMapLevel * AlphaFactor) ]

4 hours ago, sheson said:

A texture has an alpha channel. The game uses alpha testing, the alpha test threshold value is used together with the alpha test function (greater than) to determine if a pixel shows or not. So, whenever the alpha value of a pixel is greater than the default 128 threshold, it is opaque. The best value for the NiAlphaProperty of full/LOD models would be 128, because that matches the hardcoded LOD threshold and no adjustment of the texture alpha channel is required if it is added to the texture atlas.

So I assume the alpha channel of the texture should probably be made with the default threshold of > 128 (0.50196) in mind.

So alpha test function is set by NiAlphaProperty I assume, which must be > alpha test threshold in the texture data. (I constantly forget if it's actually the inverse of this statement)

If so, then NVIDIA Texture Tools default of 127 is compatible (128 - 1 to ensure mesh function is > alpha test value). I'm not sure if this is industry standard, so NiAlphaProperty should maybe default to 129 to be safe, since we don't know what software was used to create the DDS and must assume that defaults are used for alpha test threshold.

[z929669]

4 hours ago, DoubleYou said:

I notice in your snow trees they start to turn purple, which is why I said to use dilation to fix this. 

There should be no magic at threshold 127 vs 128 vs 130. The fix here is merely that the mod author failed to use alpha coverage when generating mipmaps, and you have fixed that, so as long as all thresholds are the same, the transition is seamless. 

Maybe you're right, but the value would seem to be critically dependent on DynDOLOD's test value (128) and NiAlphaproperty value (128-129) being at least one greater than texture alpha test threshold so given that, alpha test threshold of 127 seems like it could theoretically be all the difference needed to cause the texture to go null transparent or not in the mesh.

Very likely I'm wrong and testing on my part required, which I don't have the inclination at this time to do in a way that would convince myself.

[sheson]

57 minutes ago, z929669 said:

I'm confused, since this is missing a variable/coefficient and has no perns indicating order of operations [ alpha * {blank} ] = (1 + (MipMapLevel) * AlphaFactor) ]

In programming x *= y is shorthand for new-x = old-x * y and */ is always done before +-, so 1 + x * y is the same as 1 + (x * y)

for mipmapl level 1 it would be: new-alpha-pixel = old-alpha-pixel * 1 + (1 * 0.25), so new-alpha-pixel = old-alpha-pixel * 1.25

57 minutes ago, z929669 said:

So alpha test function is set by NiAlphaProperty I assume, which must be > alpha test threshold in the texture data. (I constantly forget if it's actually the inverse of this statement)

If so, then NVIDIA Texture Tools default of 127 is compatible (128 - 1 to ensure mesh function is > alpha test value). I'm not sure if this is industry standard, so NiAlphaProperty should maybe default to 129 to be safe, since we don't know what software was used to create the DDS and must assume that defaults are used for alpha test threshold.

Since I do not use NTT, I can not say what this value means for texture/mipmap creation.
The alpha threshold defined in a NIF is used by the shader to determine if a pixel shows or not when it is set to do alpha testing. The NiAlphaProperty can also change the function (less than, greater than etc.) or it can change it from alpha testing to blending. You can play around with these things in NifSkope.

The default threshold of object LOD is 128 and it always does alpha testing with greater than. If you set an alpha threshold in the LOD model of 129, then the alpha channel of the texture that is added to the atlas is adjusted to reflect that difference. For consistency it is best to set all full and LOD NIFs to an alpha threshold of 128 so there is no need to adjust textures. Then do whatever you need to in the texture tools so its alpha/transparency/mipmaps looks like whatever you want them to.

[z929669]

@Mousetick @sheson I split this conversation into it's own topic under the child forum, because it has some useful info with specific context tangential to DynDOLOD Alpha 3.0 topic but nevertheless nice to have in a single topic. I think it's pin-worthy if you guys agree.

Let me know if this splits out this discussion at the correct point or if I've otherwise missed anything else.

[Mousetick]

I've been gloating but had nothing to show for it, so here are a few shots.

In the last shot the trees on the small island with grass on the right are full models, everything else is LOD. In the other shots, I'll let you guess.

I'm still blown away by the results. I could never have imagined they were possible with these pain-in-the-neck textures. I often have to switch LODs off and on (tll) to find out which trees are LOD.

Thank you all for your help in this endeavor, particularly sheson for answering so many questions.

I'd like to recap my approach, which I'm not pretending to be a "perfect", universal solution, but it worked for me. It's not based on any "scientific" analysis, but rather on common sense deduction and the sacrifice of chickens:

• Add UseMipMaps to the name of the crown shape of the 3D tree LOD meshes.

This disables DynDOLOD's own mipmap generation and copies the mipmaps of the full branches texture to the object LOD atlas.

• Set NiAlphaProperty Threshold to 128 on the crown shape of the 3D tree LOD meshes.

This bypasses DynDOLOD's adjustment to the alpha channel of the mipmaps. To ensure continuity and consistency between full model and LOD, it's better to work on the textures and mipmaps themselves - the latter being "shared" between full model and LOD - than artificially shaping the LOD mipmaps. Also this conveniently simplifies fine-tuning of mipmap thinness and opacity by working on the full textures or the mipmap generation rather than editing the 3D LOD meshes.

• Generate mipmaps of the full branches texture with alpha-to-coverage parameterized such that the base opacity* is higher than 128, but not too high.

Ideally the transitions where there is a visible reference point should be as seamless as possible. Specifically, the transition between full texture and first mipmap level on the full model, and the transition between full model and first LOD level.

The bulk of the resulting mipmaps should have an opacity of at least 128 so that they're not clipped off when used by LOD. There will still be a few pixels with lower opacity but in my very limited experience and observations, these tend to be single pixel stragglers, or along edges, or in 1-pixel thin lines. Their loss should not be (too) noticeable at mipmap viewing distance.

Depending on the texture, using too high a base opacity for alpha-to-coverage may result in mipmaps that are too opaque, looking "coarse" and/or "solid" and/or "flat". Aside from their degraded quality, their different appearance from the full texture is noticeable in transition between full texture and first mipmap level.

* opacity = value of the alpha channel. 0 is completely transparent (invisible), 255 is fully opaque.

[Mousetick]

It may be useful to clarify that the NiAlphaProperty record in the 3D tree LOD models is meaningless and completely ineffective from the game engine perspective. Its only purpose and indirect effect is to instruct DynDOLOD, via the Threshold value, to adjust the alpha channel of its own mipmaps or of the full texture's mipmaps duplicate, so that no pixel has an alpha lower than Threshold. That's all.

In fact, when the 3D LOD model is merged into the object LOD BTO mesh, the NiAlphaProperty record is stripped out along with other fluff. There are no NiAlphaProperty records in the BTO mesh. None are necessary or possible, since LOD only uses binary alpha: less than 128 is invisible, greater or equal to 128 is fully opaque.

19 hours ago, sheson said:

The default threshold of object LOD is 128 and it always does alpha testing with greater than.

Could you please confirm strictly greater than or greater than or equal? It would seem unusual and peculiar to consider 128 as being in the lower half of a 0 to 255 range. Typically the range is divided into equal parts - 0 to 127, and 128 to 255 - and in a binary test using 128 as a threshold, 128 would evaluate to 'true'.

Thanks.

[z929669]

5 hours ago, Mousetick said:

I'd like to recap my approach, which I'm not pretending to be a "perfect", universal solution, but it worked for me. It's not based on any "scientific" analysis, but rather on common sense deduction and the sacrifice of chickens:

• Add UseMipMaps to the name of the crown shape of the 3D tree LOD meshes.

This disables DynDOLOD's own mipmap generation and copies the mipmaps of the full branches texture to the object LOD atlas.

• Set NiAlphaProperty Threshold to 128 on the crown shape of the 3D tree LOD meshes.

This bypasses DynDOLOD's adjustment to the alpha channel of the mipmaps. To ensure continuity and consistency between full model and LOD, it's better to work on the textures and mipmaps themselves - the latter being "shared" between full model and LOD - than artificially shaping the LOD mipmaps. Also this conveniently simplifies fine-tuning of mipmap thinness and opacity by working on the full textures or the mipmap generation rather than editing the 3D LOD meshes.

• Generate mipmaps of the full branches texture with alpha-to-coverage parameterized such that the base opacity* is higher than 128, but not too high.

Ideally the transitions where there is a visible reference point should be as seamless as possible. Specifically, the transition between full texture and first mipmap level on the full model, and the transition between full model and first LOD level.

The bulk of the resulting mipmaps should have an opacity of at least 128 so that they're not clipped off when used by LOD. There will still be a few pixels with lower opacity but in my very limited experience and observations, these tend to be single pixel stragglers, or along edges, or in 1-pixel thin lines. Their loss should not be (too) noticeable at mipmap viewing distance.

Depending on the texture, using too high a base opacity for alpha-to-coverage may result in mipmaps that are too opaque, looking "coarse" and/or "solid" and/or "flat". Aside from their degraded quality, their different appearance from the full texture is noticeable in transition between full texture and first mipmap level.

* opacity = value of the alpha channel. 0 is completely transparent (invisible), 255 is fully opaque.

But wouldn't this be redundant in terms of precluding DynDOLOD's adjustment, since you are using the UseMipMaps shape name in the LOD models? Setting NiAlphaProperty to 128 seems justifiable, but I didn't think it would have any impact on DynDOLOD's treatment in translating alpha on the atlas trees.

[Mousetick]

22 minutes ago, z929669 said:

But wouldn't this be redundant in terms of precluding DynDOLOD's adjustment, since you are using the UseMipMaps shape name in the LOD models? Setting NiAlphaProperty to 128 seems justifiable, but I didn't think it would have any impact on DynDOLOD's treatment in translating alpha on the atlas trees.

I'm just going by what sheson told us, there is no reason not to believe it.

I know for a fact that even when using UseMipMaps, DynDOLOD performs alpha adjustment on the mipmaps because I experienced it. When testing UseMipMaps I had initially set NiAlphaProperty Threshold to 127. Then I switched to 128 (not knowing at the time this was DynDOLOD's passthru mode for UseMipMaps) and everything went haywire because the texture mipmaps as provided by the MA were not generated with alpha-to-coverage. This is described in the OP, and in sheson's replies.

The observed behavior is consistent with what sheson stated.

There is no justification for setting to 128 other than disabling DynDOLOD alpha adjustment.

[sheson]

5 hours ago, Mousetick said:

It may be useful to clarify that the NiAlphaProperty record in the 3D tree LOD models is meaningless and completely ineffective from the game engine perspective. Its only purpose and indirect effect is to instruct DynDOLOD, via the Threshold value, to adjust the alpha channel of its own mipmaps or of the full texture's mipmaps duplicate, so that no pixel has an alpha lower than Threshold. That's all.

In fact, when the 3D LOD model is merged into the object LOD BTO mesh, the NiAlphaProperty record is stripped out along with other fluff. There are no NiAlphaProperty records in the BTO mesh. None are necessary or possible, since LOD only uses binary alpha: less than 128 is invisible, greater or equal to 128 is fully opaque.

Could you please confirm strictly greater than or greater than or equal? It would seem unusual and peculiar to consider 128 as being in the lower half of a 0 to 255 range. Typically the range is divided into equal parts - 0 to 127, and 128 to 255 - and in a binary test using 128 as a threshold, 128 would evaluate to 'true'.

Thanks.

The threshold value is used to adjust (e.g. multiply with the factor) the alpha channel of the source texture so that it still looks the same with the default alpha threshold of 128.

As already mentioned earlier, the alpha test function of LOD uses greater than, just as the default NiAlphaProperty. Exact values only really matter with uncompressed textures and if manually crafting an alpha channel that for a strange reason relies on the difference of 1 byte or 0.004 only to decide if a pixel should be transparent or not. If you need to control specific pixels exactly, use 0 and 255.

1 hour ago, z929669 said:

But wouldn't this be redundant in terms of precluding DynDOLOD's adjustment, since you are using the UseMipMaps shape name in the LOD models? Setting NiAlphaProperty to 128 seems justifiable, but I didn't think it would have any impact on DynDOLOD's treatment in translating alpha on the atlas trees.

https://dyndolod.info/Help/3D-Tree-LOD-Model
Note that LOD only has binary alpha with a fixed threshold of 128. DynDOLOD reads the alpha threshold from the NiAlphaProperty from the models used for LOD and adjusts the textures before it is added to the atlas texture. The final result in game may be to thick or thin like certain objects seem to be missing or transparency looks off. Try adjusting the NiAlphaProperty in the 3D LOD model.

Note that by default mipmaps of textures that are added to the tree LOD or object LOD texture atlas are ignored and instead are generated from the largest resolution with an alpha-to-coverage algorithm so that small details do not fade into full transparency the further away they are

These are two different things.

[z929669]

1 hour ago, sheson said:

https://dyndolod.info/Help/3D-Tree-LOD-Model

 

Note that LOD only has binary alpha with a fixed threshold of 128. DynDOLOD reads the alpha threshold from the NiAlphaProperty from the models used for LOD and adjusts the textures before it is added to the atlas texture. The final result in game may be to thick or thin like certain objects seem to be missing or transparency looks off. Try adjusting the NiAlphaProperty in the 3D LOD model.

 

Note that by default mipmaps of textures that are added to the tree LOD or object LOD texture atlas are ignored and instead are generated from the largest resolution with an alpha-to-coverage algorithm so that small details do not fade into full transparency the further away they are

These are two different things.

Thing 1: So without Shape Name "UseMipMaps", DynDOLOD reads NiAlphaProperty of say, 80, and then generates alpha-to-coverage using threshold 80? Or that Shape Name has no impact on this piece of the process?

Thing 2: This is done only if Shape Name "UseMipMaps" is NOT included in the NIF crown BsTriShape or regardless of this Shape Name being present? I assume the former from the definition:

Quote

usemipmaps - if the textures used by this shape are added to the texture atlas, use original mipmaps instead of generating them.

 

2 hours ago, Mousetick said:

I'm just going by what sheson told us, there is no reason not to believe it.

I know for a fact that even when using UseMipMaps, DynDOLOD performs alpha adjustment on the mipmaps because I experienced it. When testing UseMipMaps I had initially set NiAlphaProperty Threshold to 127. Then I switched to 128 (not knowing at the time this was DynDOLOD's passthru mode for UseMipMaps) and everything went haywire because the texture mipmaps as provided by the MA were not generated with alpha-to-coverage. This is described in the OP, and in sheson's replies.

The observed behavior is consistent with what sheson stated.

There is no justification for setting to 128 other than disabling DynDOLOD alpha adjustment.

Yes, I know you have interpreted sheson's doc and responses with confidence, but I'm still not clear with my own interpretation of the same. Call me dense, I guess.

[sheson]

54 minutes ago, z929669 said:

Thing 1: So without Shape Name "UseMipMaps", DynDOLOD reads NiAlphaProperty of say, 80, and then generates alpha-to-coverage using threshold 80? Or that Shape Name has no impact on this piece of the process?

Thing 2: This is done only if Shape Name "UseMipMaps" is NOT included in the NIF crown BsTriShape or regardless of this Shape Name being present? I assume the former from the definition:

https://dyndolod.info/Help/3D-Tree-LOD-Model
Note that LOD only has binary alpha with a fixed threshold of 128. DynDOLOD reads the alpha threshold from the NiAlphaProperty from the models used for LOD and adjusts the textures before it is added to the atlas texture. The final result in game may be to thick or thin like certain objects seem to be missing or transparency looks off. Try adjusting the NiAlphaProperty in the 3D LOD model.

There is no mention of "UseMipMaps" in this paragraph. That means it has no effect on adjusting the alpha to work with the hardcoded alpha threshold.

https://dyndolod.info/Help/3D-Tree-LOD-Model
Note that by default mipmaps of textures that are added to the tree LOD or object LOD texture atlas are ignored and instead are generated from the largest resolution with an alpha-to-coverage algorithm so that small details do not fade into full transparency the further away they are. To use the mipmaps of the original texture defined by the LOD model, add the term UseMipMaps to the BSTriShape/NiTriShape name. This should preferably be done for all LOD models using this texture. Use an unique filename for such textures, so there are no problems with other mods modifying the same vanilla filenames.

This paragraph explains what "UseMipmaps" does. It instructs DynDOLOD to not generate mipmaps but use the ones from the texture. Note how this paragraph does not mention the alpha threshold or NiAlphaProperty.

Logic dictates that if the alpha channel needs to be adjusted, it always happens to all mipmap levels regardless their source.

[z929669]

1 hour ago, sheson said:

https://dyndolod.info/Help/3D-Tree-LOD-Model
Note that LOD only has binary alpha with a fixed threshold of 128. DynDOLOD reads the alpha threshold from the NiAlphaProperty from the models used for LOD and adjusts the textures before it is added to the atlas texture. The final result in game may be to thick or thin like certain objects seem to be missing or transparency looks off. Try adjusting the NiAlphaProperty in the 3D LOD model.

There is no mention of "UseMipMaps" in this paragraph. That means it has no effect on adjusting the alpha to work with the hardcoded alpha threshold.

https://dyndolod.info/Help/3D-Tree-LOD-Model
Note that by default mipmaps of textures that are added to the tree LOD or object LOD texture atlas are ignored and instead are generated from the largest resolution with an alpha-to-coverage algorithm so that small details do not fade into full transparency the further away they are. To use the mipmaps of the original texture defined by the LOD model, add the term UseMipMaps to the BSTriShape/NiTriShape name. This should preferably be done for all LOD models using this texture. Use an unique filename for such textures, so there are no problems with other mods modifying the same vanilla filenames.

This paragraph explains what "UseMipmaps" does. It instructs DynDOLOD to not generate mipmaps but use the ones from the texture. Note how this paragraph does not mention the alpha threshold or NiAlphaProperty.

Logic dictates that if the alpha channel needs to be adjusted, it always happens to all mipmap levels regardless their source.

Thanks but replying with the doc text doesn't help me much. I was looking for explicit answers to my questions. I think I understand most of this, but could your reply directly to the following(?):

Does DynDOLOD read NiAlphaProperty of say, 80, and then generate alpha-to-coverage using threshold 80?

[Mousetick]

9 hours ago, sheson said:

As already mentioned earlier, the alpha test function of LOD uses greater than, just as the default NiAlphaProperty.

This doesn't quite answer what I was trying to ask. Let's start over, if you don't mind. When you say:

Quote

The default threshold of object LOD is 128 and it always does alpha testing with greater than.

The BTO mesh doesn't have any NiAlphaProperty with operator flags or threshold, so we don't know if it's greater than or greater than or equal, or if the threshold is 127 or 128.

But if you what you say is true, then that means the minimum alpha value for a LOD pixel to be opaque is 129 and 128 is transparent/invisible. I find that highly unlikely and unusual. But if it is true, then this has huge implications that ripple down the whole chain. Starting with DynDOLOD adjustment of mipmaps alpha: the correct "neutral" value should be 129, not 128. Etc...

9 hours ago, sheson said:

The threshold value is used to adjust (e.g. multiply with the factor) the alpha channel of the source texture so that it still looks the same with the default alpha threshold of 128.

How does one go from a threshold to a multiplication factor? Is DynDOLOD performing some kind of linear interpolation on the entire mipmap alpha channel? How does it matter if the alpha channel is linearly interpolated or not with LOD? The LOD response to alpha input is not linear, it's binary.

But you mentioned "alpha channel of source texture", so I guess the linear interpolation is applied before DynDOLOD creates the mipmaps, which would make sense. What happens exactly in the case of UseMipMaps?

Thanks