xLODGen - Terrain LOD beta 129 for FNV, FO3, FO4, FO4VR, TES5, SSE, TES5VR, ENDERAL, ENDERALSE

[sheson]

Thanks to all your hard work, I was able to generate some VR agreeable Tamriel terrain and redo the object meshes.

 

https://www.nexusmods.com/skyrimspecialedition/mods/16609

Great. That needs drastically improved object LOD from DynDOLOD though :) I can never believe how bleak vanilla looks and how so many important objects are missing from LOD.

Edited April 22, 2018 by sheson

[godescalcus]

Hi, Sheson and folks, I'm still generating assets for my first test at Skyrim SE, but I'd like to confirm what someone told me, that you can create a dummy esp and pack the output files as a bsa archive? The only procedure I know for doing this is the one explained by Prometheus here at afkmods.com. That was 2016 and SSE had just been released - is there a more convenient alternative for SSE, similar to BSAOpt?

 

Then, if I do pack everything into a BSA and load it last (after DynDOLOD?), won't those assets still be overridden by any loose files I might have from other mods? I'm thinking especially of flora and landscape texture mods, although my fix for landscape textures is Abendaron's Simply Optimized Textures for SSE, which is also packed as a bsa and loaded right at the top; but also original mods for new worldspaces if those mods aren't packed into bsa as well. Examples of this would be Wyrmstooth 1.17A, Quaxe's Questorium or Beyond Reach 3.999 (the latter has a bsa but terrain textures are loose files).

 

DynDOLOD itself is loose files but I suppose it would me possible to pack it into bsa as well and manage load order with SSELODGen. But to do the same to my whole load order... Yikes.

[sheson]

Hi, Sheson and folks, I'm still generating assets for my first test at Skyrim SE, but I'd like to confirm what someone told me, that you can create a dummy esp and pack the output files as a bsa archive? The only procedure I know for doing this is the one explained by Prometheus here at afkmods.com. That was 2016 and SSE had just been released - is there a more convenient alternative for SSE, similar to BSAOpt?

 

Then, if I do pack everything into a BSA and load it last (after DynDOLOD?), won't those assets still be overridden by any loose files I might have from other mods? I'm thinking especially of flora and landscape texture mods, although my fix for landscape textures is Abendaron's Simply Optimized Textures for SSE, which is also packed as a bsa and loaded right at the top; but also original mods for new worldspaces if those mods aren't packed into bsa as well. Examples of this would be Wyrmstooth 1.17A, Quaxe's Questorium or Beyond Reach 3.999 (the latter has a bsa but terrain textures are loose files).

 

DynDOLOD itself is loose files but I suppose it would me possible to pack it into bsa as well and manage load order with SSELODGen. But to do the same to my whole load order... Yikes.

Read BSAs and You

 

Loose files always overwrite files in BSAs, so packing LODs into BSA will require some work to make sure the LOD meshes/textures are not overwritten by loose files.

 

LOD meshes and textures are always contained in meshes/terrain/[worldspace] and textures/terrain/[worldspace], however object LOD also uses textures from other paths - for example textures\landscape\mountains\MountainSlab02.dds

 

The best tool to create BSAs for any game is the command line tool BSArch from Zilav. If you require a GUI, use Archive.exe that comes with CK.

 

I do not recommend packing the generated LOD into BSAs to avoid overwrite issues and use a mod manager to properly deal with loose files and load orders.

[alt3rn1ty]

I have not tried it yet, but I have very few loose files these days (none which conflict with any output files from xLODGen or DynDOLOD) and intend having a dummy plugin to load late, and chain load all output files from a BSA. BSArch would also be my preferred method and use its compression switch -z for faster throughput when loading.

[godescalcus]

I have not tried it yet, but I have very few loose files these days (none which conflict with any output files from xLODGen or DynDOLOD) and intend having a dummy plugin to load late, and chain load all output files from a BSA. BSArch would also be my preferred method and use its compression switch -z for faster throughput when loading.

Thanks for the input, I'll be using BSArch in the future, for sure. I've been trying hard to get to a more bsa'd build and I sure did notice a performance impact when moving from mostly loose files to mostly bsa. But some mods like Wyrmstooth and the like do come in loose files... To go over my whole mod list and pack every mod would be a huge task... Nonetheless, it seems practicable to check the lodgen files for conflicts before packing it, then checking whether it's possible to pack also the earlier loading mods that have conflicting loose files. But it could sprawl from there if those mods, in turn, had other conflicts that needed to be looked after, and then those might have other conflicts as well... The only sure way I see to make this work is to plan your build from the start to have only archived mods, and no loose files, or as little as possible, either sacrificing or processing mods that have them. Whew!

[icecreamassassin]

Alrighty so one more pass at this :D I have been trying for several days to fiddle with the various settings and so far I have found no way to compensate for the Noise.dds effects which don't totally wash out the LOD landscape textures. Editing the Noise.dds file for testing is fine, but it's not a viable solution really for release since it requires altering a file that is used universally by the vanilla game and other modded worldspaces as well as general landscape mods like Vivid Landscapes, so I am hoping that someone will have some input on how to adjust the settings. I have been fiddling so much I think I've confused myself. Based on Sheson's recommended "starting point" of setting detail of LOD4 to 10 and increase it by 5 for each LOD level beyond that, that would seem that LOD32 is the closest area of LOD which needs higher detail closer to the actual textures? Do I have that understood properly? I've tried fiddling with the gama and brightness and ultimately I don't seem to be coming up with anything that is masking out the horrible noise.dds. This isn't a huge issue over darker textures but I am working with sand in many areas and the result looks like crap no matter what setup I use. The distant sand looks almost like dirt with a little bit of sand in it rather than sand with a little bit of dirt (thanks to noise.dds). So what would folks recommend I try other than editing the DDS? Thanks.

 

Oh wait.... I just realized I got the quality setting reversed; lower = better my bad. I'll fiddle one more pass and see what I get

Edited April 24, 2018 by icecreamassassin

[alt3rn1ty]

Oh wait.... I just realized I got the quality setting reversed; lower = better my bad. I'll fiddle one more pass and see what I get

Not sure if this will help with your sand, but I have a tip maybe not so many will know of these days :

 

A much better noise.dds than vanilla

 

An old mod for Skyrim LE, which has a noise.dds which is still better than Skyrim SE

 

Detailled Terrain and Tree LOD

 

I dont use the tree LOD files, just the noise.dds (the idea with the tree LOD billboards here was to add a bit of grass to the base of them, which initially seems like a good idea, but imagine an L shape, the trunk of the tree being the upright, and the grass being the foot of the L, now stick that L on a slope of a mountain edge you see in the distance \L = grass in mid air) ..

 

.. But Rennn did a pretty good job on the noise.dds. Here's a comparison with SSE, and I think the overall impression is a little lighter than vanilla too, seems to work well for me with xLODGen anyway

 

 

 

Read the mod description, it can be used as a resource ;)

Edited April 25, 2018 by alt3rn1ty

[icecreamassassin]

Thanks, but I have a modified noise.dds that Sheson suggested the color scheme for and it works great but I have concerns about packing it because it affects all worldspaces. That'd be like doing an edit to Ulfric's NPC record rather than having scripts do what you need to him; too many potential conflicts. I'm trying to use gama, brightness and contrast settings in place of altering the noise.dds.

[zilav]

I have not tried it yet, but I have very few loose files these days (none which conflict with any output files from xLODGen or DynDOLOD) and intend having a dummy plugin to load late, and chain load all output files from a BSA. BSArch would also be my preferred method and use its compression switch -z for faster throughput when loading.

Offtopic, but due to zlib's limited decompression throughput (less than SATA HDD speeds and way less than PCIe SSD) you'd better go for decompressed archives if you are looking for speed and reduced stuttering. The only exception is Skyrim Special Edition where they use lz4 instead of zlib.

[alt3rn1ty]

So by exception do you mean BSArch does use LZ4 for SSE (you say "where they use LZ4" which makes me think you might not be), or is it using slow zlib throughput for all games?

Going by BSArch description all game flags are described as using the correct format per game .. But compression method is not mentioned, so I am not clear on what is being used and just assumed the new LZ4 was being used for SSE which is desireable for the greater throughput.

 

Edit : Sorry Sheson for continuing off topic here, this will be my last.

 

 

Thanks, but I have a modified noise.dds that Sheson suggested the color scheme for and it works great but I have concerns about packing it because it affects all worldspaces. That'd be like doing an edit to Ulfric's NPC record rather than having scripts do what you need to him; too many potential conflicts. I'm trying to use gama, brightness and contrast settings in place of altering the noise.dds.

 

 

Ah understood, this is for a mod project and you dont want to introduce such a sweeping change to everyones setup.

 

I also dabbled with those settings a bit but its awkward to even guess a rough approximation of what the outcome may be.

Edited April 25, 2018 by alt3rn1ty

[sheson]

So by exception do you mean BSArch does use LZ4 for SSE (you say "where they use LZ4" which makes me think you might not be), or is it using slow zlib throughput for all games?

The BSA version implies the compression format. The BSA version 0x69 for Skyrim SE always uses LZ4. For Skyrim it is 0x68 always uses zlib. (Unless the flag for XMem for XBOX 360 is set, which nobody cares about)

 

So LZ4 is used for SKyrim SE.

Edited April 25, 2018 by sheson

[alt3rn1ty]

Ace! :), LZ4 is superb, and personally SSE is all I am bothered about these days so BSArch is weapon of choice. And for FO4 the biggest concern is being able to produce the special textures archive for optimal performance = BSArch also has that covered.

 

I wonder if Hard drive technology has now taken over in terms of speed versus what BSAOpt used to offer as a faster throughput for older games archives, one of the guides I duplicated from old bethesda forums includes a calculation Ethatron once did versus HD technology at the time .. Which is quite a few years ago now. From the "BSAs and You" topic on AFK Mods :

 

 

 

4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?

To quote Ethatron ( Via the Wayback machine, original site is down ) NIFOpt Documentation :
 

Ethatron said

  Ethatron said:

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

• Harddisk-throughput: 20MB/s
• Memory-throughput: 2000MB/s
• zlib-throughput: 80MB/s
• uncompressed size: 100MB
• compressed size: 60MB

This yields:

• Reading 100MB at 20MB/s from disk takes 5s.
• Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.

In case the data is very small it's probable that seeking time will start dominating the times:

• Reading 1MB at 20MB/s from disk takes 0,05s.
• Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.

Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

 

So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs. 

 

 

[godescalcus]

Tamriel with @alt3rn1ty's settings sort of, quality 5,10,10,10

 

It replaced "better falskaar and wyrmstooth maps with roads" and the result is much better, I only seem to be missing the roads... Is there a way to add them? Like an overlay?

 

Edited April 26, 2018 by godescalcus

[sheson]

I only seem to be missing the roads... Is there a way to add them? Like an overlay?

Not automatically with xLODGen. You would have to hand paint them onto the textures just like it was done in the mod with the vanilla LOD textures.

 

You could try using DynDOLODs road object LOD and changing the mesh rule to add them to LOD 16 as well, but it most likely will not work well because the coarse terrain LOD meshes will swallow them up most of the time.

Edited April 26, 2018 by sheson

[godescalcus]

Not automatically with xLODGen. You would have to hand paint them onto the textures just like it was done in the mod with the vanilla LOD textures.

That would be a nice project, but not an easy one, it seems. Would have to be able to overlay two montages, one with xLODGen's textures and another with "better maps with roads" and trace the roads onto the textures. Seems like it could be done with layers in Photoshop but it beats anything I have done so far.

 

Are the texture tiles always the same? I'm wondering if a set of masks could be made from "better maps" with only the roads and a batch that applied to each tile the corresponding mask, something one could run after each terrain lod update.

Edited April 26, 2018 by godescalcus