TexGen/DynDOLOD out of graphics memory Linux

[Vraylle]

I'm attaching the realtime log, as that's the only one I can generate.

I'm running TexGen for SSE under Proton 9.0.4 on Linux. The PC has 64GB RAM and 8GB swap.

After scanning the load order, TexGen displays "Gathering base records for billboards". Then memory usage (RAM and SWAP) go to 100% and the entire system freezes. After about 15 minutes, the entire Wine prefix dies/crashes. Memory is recovered but the swap remains about half full. I can see from the realtime log that it seems to be doing work, but since it uses everything it looks like the OS is killing the process when it gets the chance (again, about 15 minutes later).

I've been looking for a way to restrict the memory available to that process to just let it run and see what happens, but haven't had any luck yet.

I will say this is the first time I've run it on this particular PC, but I've run it many, many times prior.

[sheson]

7 hours ago, Vraylle said:

I'm attaching the realtime log, as that's the only one I can generate.

I'm running TexGen for SSE under Proton 9.0.4 on Linux. The PC has 64GB RAM and 8GB swap.

After scanning the load order, TexGen displays "Gathering base records for billboards". Then memory usage (RAM and SWAP) go to 100% and the entire system freezes. After about 15 minutes, the entire Wine prefix dies/crashes. Memory is recovered but the swap remains about half full. I can see from the realtime log that it seems to be doing work, but since it uses everything it looks like the OS is killing the process when it gets the chance (again, about 15 minutes later).

I've been looking for a way to restrict the memory available to that process to just let it run and see what happens, but haven't had any luck yet.

I will say this is the first time I've run it on this particular PC, but I've run it many, many times prior.

Did you change MaxTextureSize in TexGen_SSE.ini? Change it back to 16384 or try lowering it to 8192.

[Vraylle]

11 hours ago, sheson said:

Did you change MaxTextureSize in TexGen_SSE.ini? Change it back to 16384 or try lowering it to 8192.

I had not changed it, but did try lowering it to 8192 as you suggested. It seemed to take a second longer to eat all memory and freeze the system.

Just for grins, I lowered it to 256 and tried again...same result, and about as quickly as 8192 did.

[sheson]

39 minutes ago, Vraylle said:

I had not changed it, but did try lowering it to 8192 as you suggested. It seemed to take a second longer to eat all memory and freeze the system.

Just for grins, I lowered it to 256 and tried again...same result, and about as quickly as 8192 did.

In that case try changing MaxMultiSamples to 8 or 4.

[Vraylle]

4 hours ago, sheson said:

In that case try changing MaxMultiSamples to 8 or 4.

OK, set MaxTextureSize to 8192 and MaxMultiSamples to 4. This time it ate all the memory and froze the system for a few seconds then crashed. Again no logs except the realtime log. It looks very similar except it has 11576 lines instead of 2891. Too large to send to paste.ee.

[sheson]

2 hours ago, Vraylle said:

OK, set MaxTextureSize to 8192 and MaxMultiSamples to 4. This time it ate all the memory and froze the system for a few seconds then crashed. Again no logs except the realtime log. It looks very similar except it has 11576 lines instead of 2891. Too large to send to paste.ee.

Read the first post, my signature or https://dyndolod.info/Official-DynDOLOD-Support-Forum#Post-Logs how to zip and upload large log files to a file service.

[Vraylle]

8 hours ago, sheson said:

Read the first post, my signature or https://dyndolod.info/Official-DynDOLOD-Support-Forum#Post-Logs how to zip and upload large log files to a file service.

The file service link there hasn't worked for me in the last two days (website seems to behave normally, but uploads never complete).

Uploaded to an alternate service.

[sheson]

On 4/15/2025 at 3:06 PM, Vraylle said:

The file service link there hasn't worked for me in the last two days (website seems to behave normally, but uploads never complete).

Uploaded to an alternate service.

Use this test version of TexGen https://mega.nz/file/9QRCWKhT#Oa7KmQjKNXkjPIZd_wWDQv2uYPkP0Rvjx5reKquJJQI

Add MaxRenderResolution=4096 to TexGen_SSE.ini under [TexGen] to see if it makes a difference.

[Vraylle]

On 4/16/2025 at 8:26 AM, sheson said:

Use this test version

Add MaxRenderResolution=4096 to TexGen_SSE.ini under [TexGen] to see if it makes a difference.

Behavior was much the same (with both the test file and the INI change you mentioned), but it seemed to immediately crash when maxing out the memory, rather than have a lengthy delay.

New realtime log.

 

[sheson]

13 hours ago, Vraylle said:

Behavior was much the same (with both the test file and the INI change you mentioned), but it seemed to immediately crash when maxing out the memory, rather than have a lengthy delay.

New realtime log.

It looks like something went wrong adding MaxRenderResolution=4096 under [TexGen] in TexGen_SSE.INI

Make sure to copy and paste the setting and there are no extraneous spaces anywhere.

The debug log line [TwbRender.Create] <Debug: Max render resolution: 32768 x 32768> should show 4096 then.

[Vraylle]

54 minutes ago, sheson said:

It looks like something went wrong adding MaxRenderResolution=4096 under [TexGen] in TexGen_SSE.INI

Make sure to copy and paste the setting and there are no extraneous spaces anywhere.

The debug log line [TwbRender.Create] <Debug: Max render resolution: 32768 x 32768> should show 4096 then.

Quite right, I completely misread that before...apologies. No log from the run because it worked, and quite quickly. Never went above 20% memory usage, in fact. Would it be worth trying a higher value, or is the visual difference negligible?

Edited April 17 by Vraylle

[sheson]

3 hours ago, Vraylle said:

Quite right, I completely misread that before...apologies. No log from the run because it worked, and quite quickly. Never went above 20% memory usage, in fact. Would it be worth trying a higher value, or is the visual difference negligible?

The value does not need to higher than the largest texture TexGen has to render. Which typically won't be over 2048. Nothing should realy be over 4096 unless base sizes are greater than 1024 (4320p).

[Vraylle]

Now I'm seeing a crash on the actual LOD generation. It looks like it completes the LOD itself (I have 12.1GB of meshes) but crashes pretty immediately on the textures (17 files, 1.4MB). The textures folder contains the "lod" subfolder with those 17 files. There are no DynDOLOD or Terrain subfolders. The main output window vanishes when the command-line LOD gen starts, but I feel like it used to stay visible. The command-line window closes when the LOD generation is (I think) complete, and that's the end. No errors or logs, just quits.

No log files are generated. (no bugreport.txt, DynDOLOD_SSE_[debug?]_log.txt, etc.)

Memory and CPU usage is around 20% for the entire process, except for one spike right before beginning the actual LOD generation.

I have modified DynDOLOD_SSE.ini to use fewer cores on several options (between 4 and 8 cores on a 16-core CPU) and set MaxTextureSize to 4096.

Any suggestions?

Edited April 18 by Vraylle

[sheson]

On 4/18/2025 at 4:53 AM, Vraylle said:

Now I'm seeing a crash on the actual LOD generation. It looks like it completes the LOD itself (I have 12.1GB of meshes) but crashes pretty immediately on the textures (17 files, 1.4MB). The textures folder contains the "lod" subfolder with those 17 files. There are no DynDOLOD or Terrain subfolders. The main output window vanishes when the command-line LOD gen starts, but I feel like it used to stay visible. The command-line window closes when the LOD generation is (I think) complete, and that's the end. No errors or logs, just quits.

No log files are generated. (no bugreport.txt, DynDOLOD_SSE_[debug?]_log.txt, etc.)

Memory and CPU usage is around 20% for the entire process, except for one spike right before beginning the actual LOD generation.

I have modified DynDOLOD_SSE.ini to use fewer cores on several options (between 4 and 8 cores on a 16-core CPU) and set MaxTextureSize to 4096.

Any suggestions?

Read https://dyndolod.info/Official-DynDOLOD-Support-Forum#Post-Logs how to enable the realtime log for DynDOLOD and upload it.

Download this test version of DynDOLOD https://mega.nz/file/oIhEQaCZ#d1FvBGM9fFFHqOGHHVTd6X0BaI4TBCpp_Uj6pnklA1M
Add MaxRenderResolution=4096 under [DynDOLOD] to DynDOLOD_SSE.ini and report if it makes a difference.

[Vraylle]

14 hours ago, sheson said:

Read https://dyndolod.info/Official-DynDOLOD-Support-Forum#Post-Logs how to enable the realtime log for DynDOLOD and upload it.

Download this test version of DynDOLOD https://mega.nz/file/dQYnTRoB#czPPP8mkKmsMwAatOJaPi90CG9RsGMNsOuUwHDdMbv8
Add MaxRenderResolution=4096 under [DynDOLOD] to DynDOLOD_SSE.ini and report if it makes a difference.

Not sure which part did the trick, but it seemed to complete successfully and the main windows stayed "alive". The first run froze about a minute in, but the second ran all the way through. Thanks again!

[Strigifowl]

After recently getting back into modding did I of course install the latest version of Dyndolod 3. However, for whatever reason when I attempt to run Texgenx64.exe it crashes without generating any logs when it attempts to load in billboard records. I also had an issue with running DynDOLODx64 but I solved it by adding LockTexconv = 1 to it's .ini however doing the same for texgen's .ini file didn't help. Also to note is that my CPU and RAM usage spikes up heavily right before Texgen crashes.