Linux/Proton: A call to an OS function failed

[itsdnoftheworld]

Hello, I have been having an annoying issue running Dyndolod 3 on my Proton setup related to texconv. While running Dyndolod after successfully generating xlodgen and texgen, I am unable to proceed with generating Dyndolod, as I eventually get a message saying "Could not execute "Z:\home\dillon\Games\DynDOLOD\Edit Scripts\Texconvx64.exe" -nologo -gpu 0 -y -m 1 -aw 256 -f R8G8B8A8_UNORM -o "C:\users\steamuser\Temp\DynDOLOD_SSE" "C:\users\steamuser\Temp\DynDOLOD_SSE\379794EC272A4EF98BAF93F0B0726085.dds". Then I get a pop-up about Call to an OS Function Failed. No .esp is mentioned.

Since I am on Linux, I don't have UAC or AV on my computer, and after changing permissions of some of the folders, specifically my game folder, mods folder, and the Dyndolod folder I have been able to run it once or twice. However, after changing a texture mod, I have been completely unable to get around this Texconv issue. I believe that Texconv should be running, since I can see it in the task manager and I can see some .dds files being converted. I also was able to run texgen and xlodgen with no issue. In fact I have run Dyndolod successfully earlier today! I have tried placing Dyndolod folder in both my Z: drive (the native Linux system) and my C: drive (the emulated Windows drive), and both run into the same error. I also reinstalled and repaired dotnet and C++, and I was able to run texconv (the official one from Github) with the below output. I also was able to convert images into .dds from the command line. However I could not replicate this with the one downloaded alongside Dyndolod. Is this expected behavior?

I am currently unable to test on Windows since my install is all messed up, but I am just looking to see if there is anything I have missed in troubleshooting. Any advice would be greatly appreciated.
 

Bug Report: https://pastebin.com/B3Te9XG4

SSE Log: https://paste.ee/p/Hhg78

Texconv Test: https://paste.ee/p/6vtVc

[sheson]

On 12/18/2024 at 7:59 PM, kontotnok said:

Got a successful full run! I generated Objects, Trees and Terrain LOD!!

Here's the logs: https://filebin.net/6crrb0appzm9987d

If you want me to keep testing stuff I'll gladly do so!

If you like, redo the test with the released xLODGen beta 129.

In particular it would be interesting to know if LODGenx64.exe compiled for .Net Framework 4.8 is still having issue unlike the .NET 6 version.

[MagneticIce]

I myself have been carrying out a fair amount of testing around the "A call to an OS function failed" error so I figured I would share my findings.

Firstly, this is specifically an issue with Texconv and as such it also effects other software that uses Texconv such as Wabbajack.

Secondly, this error only occurs on versions of wine with the staging patches applied to it (i.e. wine-staging, proton etc.). The basic version of wine without the staging patches can be used to run both Texconv and Dyndolod without issue (at least as of wine version 9.22).

Thirdly, it seems like this error may only occur on specific distributions of Linux but I haven't done enough testing to confirm this.

Fourthly, this error occurs at random. I tried running Texconv on the same texture file over 200,000 times using a bash script and found that, with a probability of around 1 in 4000, Texconv would fail and would not produce any output. When Texconv failed, the following attempt to run Texconv always succeeded so I suppose one failure doesn't affect the odds of another failure occurring.

[CornyTrace]

On 12/18/2024 at 10:09 AM, sheson said:

Do the same with this test version https://mega.nz/file/EZBkhDzK#Y134Dju7ZsZqpC3utCM9i68vXcQ0kltifay5_XHgzWk

Hi, I have the same problem and would like to test this version as well, unfortunately the link is no longer available. Could you please reupload it?

[sheson]

15 hours ago, CornyTrace said:

Hi, I have the same problem and would like to test this version as well, unfortunately the link is no longer available. Could you please reupload it?

That test version does not exist anymore since it is older than the latest release version which includes the changes.

[CornyTrace]

22 hours ago, sheson said:

That test version does not exist anymore since it is older than the latest release version which includes the changes.

It says on nexusmods that dyndolod 3 182 was uploaded on 11 oct 2024, while your test version post was made on 18 dec 2024, I have also not experienced texgen being retried if it failed with dyndolod 3 182.

[sheson]

1 hour ago, CornyTrace said:

It says on nexusmods that dyndolod 3 182 was uploaded on 11 oct 2024, while your test version post was made on 18 dec 2024, I have also not experienced texgen being retried if it failed with dyndolod 3 182.

The post you linked, the test version and all discussion between kontotnok and me are about xLODGen and generating LOD for FNV.

For a test version of TexGen, see https://stepmodifications.org/forum/topic/20153-error-opengl-framebuffer-objects-unsupported/page/2/#findComment-282545

However, pay attention to newer and folder posts. The "OS" having issues can have plenty of reasons. The reason for the "OS" having issues should be fixed at source if possible.

[ssuamier1]

I have created  a guide mainly for myself but also for having a documentation how to setup DynDOLOD in a stable Proton environment.
This is a guide for people with a good understanding on how to setup tools with Proton and Mod Organizer (for more details look for YT Videos and Beginner Guides).

The guide also shares my experiences with that setup. I run into two types Texconv errors regulary, but all these looked random.
With a lot of patience I was able to get a complete Dyndolod Output generated, but only using single parts and keeping the generated files.

The only issue with splitting up is that the generated Dyndolod / Occlusion plugins are probably incomplete (comparing the size of the plugins, I have not checked in SSEEdit).

Issues:
1. In the first phase I see "Errors" but mainly on Tree ressources (both DynDOLOD and from mods). I'm not 100% sure, but it looks like no duplication appeared, so it feels "random".
   I was able to recover after 1-6 retries. Afterwards DynDOLOD goes into second phase of actual LOD generation.
2. In the second phase I rarely had "Errors". These were always fatal and I had these usually around 20-30min (40min max, I never was able to fully complete it).

Apart from the question why in comparison to time estimations around 15min on Windows for me DynDOLOD run much slower (around > 1h) I had always issues with Texconv if I understand the logs correctly. I have not exactly tracked which textures failed, but I had the same effect that on a second or third run a single generation was finishing.

 

Spoiler

Guide based on Experience with DynDOLOD 3.0 - 182
=================================================

This Guide was created and published by ssuamier on Jan 2025.

Disclaimer: This guide is meant as a help to setup and run the above
Version of DynDOLOD with official stable Proton 9.0.4 / Linux using
Mod Organizer 2.5.2 as base. Before using this guide read the official
documentation how to setup DynDOLOD. This guide assumes that everything
has been setup. This guide is not meant as a replacement for the official
documentation. In doubt, the official documentation is always correct.

Intro: While setup itself is straight forward with MO2, I had to face
a lot of issues until I was able to get a working Dyndolod Output.
This guide is meant as a help for you to speed up the process.

Requirements: Minimal 32 GB of memory, Recommended 64 GB of memory.
A very fast SSD (type NVME) with at least 30 GB free space (rule of
thumb: make sure there are more than 20% free space). A Good CPU is a plus.

Newer releases of DynDOLOD, Proton and MO2 may render parts of this Guide
as invalid, especially in regards to memory consumption and "Errors".

Guide:
------

I    Create a new prefix after backing up the old one in your steam folder
     where (Skyrim) is located.
II   Backup your base game and rename the game folder in "common"
III  Use proton 7.0 as compatibility in Steam and start the game once.
IV   Steam will download the latest game and may generate Vulkan Shaders.
V    Quit the game and start protontricks or winetricks on your prefix
VI   Install dotnet48, vcrun2022, dotnetdesktop as per official documentation
     using winetricks / protontricks
VII  Go back to steam and change proton to version 9.0.4
     (Note: Using "protonfixes" you should be able to install dotnet40 on version 9.0.4)
VIII Run either winetricks or the game to update your prefix to 9.0.4
IX   Run MO2 and load your DynDOLOD Profile. If you didn't set it up,
     deselect all mods that are not relevant for LOD creation (except base)
X    Check that xlodgen, texgen and dyndolod are working
XI   Use/Create a shell script to start MO2. Add a parameter that starts DynDOLOD*.
XII  (Optional) create a script shortcut for your desktop. Consider to switch to
     a desktop temporarily that takes less ressources than Gnome/KDE/Plasma
XIII Restart your computer
XIV  Setup DynDOLOD with MO2 and prepare dyndolod (ressources + xlodgen + texgen)
     as per official documentation
XV   Check that you have rights to write into the Dyndolod folder
XVI  (Optional) Change DynDOLOD_SSE.ini to start DynDOLOD in Advanced Mode
XVII (Optional) Enable your swap file/partition**
XVIII Start DynDOLOD and set it up (as per official documentation)
XVIX Run DynDOLOD with all parts first. If you encounter a lot of "Errors"
     you can consider to use the steps described below
XX   Quit in your prefered way, I recommend to create a zip file.

Extra Steps in case of a lot of "Errors":
-----------------------------------------

a)   Select one part
b)   Generate LODs.
c)   Rerun ("Restart") when necessary
d)   Restart computer when you feel it is getting slower
e)   After creating zip file, extract it in DynDOLOD_Output folder
f)   Rename the zip file. Move it to a safe place.
g)   (Optional)After each part is finished, check the logs for any
     problems that you may have overlooked. Backup logs when in doubt
h)   After your final run you need to create the DynDOLOD plugins.
     Currently the only way I'm aware of doing this, is to merge the
     each Dyndolod plugin created previously with tool like Wrye Bash.

* The script can looks like this, but you need to adjust paths etc:

--------------- cut here ---------------
#!/bin/bash

export STEAM_COMPAT_CLIENT_INSTALL_PATH=/path/to/.local/share/Steam
PROTON_FORCE_LARGE_ADDRESS_AWARE=1 PROTON_NO_ESYNC=1 STEAM_COMPAT_DATA_PATH=/path/to/steam/folder/steamapps/compatdata/489830 /path/to/steam/steamapps/common/Proton\ 9.0\ \(Beta\)/proton waitforexitandrun /path/to/Modding/MO2-Skyrim/ModOrganizer.exe 'moshortcut://DynDOLODx64'
--------------- cut here end -----------

** If RAM is an issue even with swap or there are reasons you cannot use
swap but run out of memory, you may want to not use MO2 which can save
up to 1GB of memory. In this case, start MO2, open an explorer window and
copy your data folder to a different place. Outside MO2 copy the content
temporarily into your game folder and run DynDOLOD without MO2.

Experience Information
----------------------

1. Memory consumption: up 15 GB in my setup with around 300 relevant mods.
   There can be peaks. Linux, the desktop, MO2 etc. will take several
   GB of RAM as well.

2. Speed: For my AMD 5600G CPU I have up to 15min for all mods in the
   first phase. The second phase takes > 1 hour.

3. Speed splitted: On my system I had > 5 hours to generate all parts.
   Dyndolod shows 18 parts that can be processed

4. Stability: First phase is the most unstable. I had up to 6 restarts
   but I did not observe one problem multiple times. I'm not 100% sure
   if there were even any duplicates reported within all of the runs.
   
5. Memory Fragmentation: I noticed that after a few runs my desktop gets
   slow and the number of restarts seem to increase. I rebooted to reset
   the memory fragmentation.
   
6. Swap: I use additional swap to deal with spikes. I could not observe
   any changes in the observed behaviour of 5. by switching off swap.
   While there is a performance impact it was limited to short periods.

7. Stabiliy II: The number of problems in the second part was considerably
   lower. The main problem is that DynDOLOD has no way to retry / continue
   after an "Error" which can lead to a massive amount of time to finish
   DynDOLOD Output.

8. Merge: If splitting up the (in my case 18) parts, Dyndolod creates
   does only include the part in the ESP/ESM plugins. I don't know if
   merging (Wrye Bash / ZMerge) of the single parts can be avoided.

9. Proton Versions: I didn't notice a difference between Proton GE and
   Proton Stable. The "Error" problem did not resolve by using stable version.
   Also additional options for Proton did not seem to have any influence.

10. Why use Proton 7? dotnet48 does not install on Proton 9 because auf
  dotnet40 does not install on Proton 9. On the other hand MO2 2.5.x
  requires Proton 9 (it will not start on older releases). If you are
  using the older MO2 2.4 release, you can try Proton 7 if it brings
  more stability to DynDOLOD.

In addition to this: I have no issues in running xLODGen and TexGen (I run
it a few times since I had not installed all mods dependencies)

The DynDOLOD Mod Author mentioned that a test version did a "simple" retry
of LOD generation that worked. The true reason why the previous generation
failed was not clear at that point.

These logs are from my latest run with all parts enabled. The "Error" in this case was on a tree texture, but it was a different one than on the previous run.
https://filebin.net/qckm9psnivewi6r0

[ssuamier1]

I try to create dynamic LODs for Skyrim SE with DynDOLOD 3 Alpha 182 but I have random Errors with Texconv during generation.
Many of them are related to tree textures (from different mods). I have checked the textures but could not find any errors in them.

For example:

[01:26] <Error: Could not execute "D:\Modding\tools\DynDOLOD\Edit Scripts\Texconvx64.exe" -nologo -gpu 0 -y -m 1 -aw 256 -f R8G8B8A8_UNORM -srgb -o "C:\users\steamuser\Temp\DynDOLOD_SSE" "C:\users\steamuser\Temp\DynDOLOD_SSE\13E47B1E29F449EEAFAE3EA90369C1C3.dds">
[01:47] A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]
A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]
Error: A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]

I could recover from such errors via restart (and they do not reappear in the next run).
Later I got a kind of fatal error:

[21:58] [DLC2SolstheimWorld] Creating texture atlas D:\Modding\tools\DynDOLOD\DynDOLOD_Output\textures\DynDOLOD\LOD\DynDOLOD_DLC2SolstheimWorld.dds 8192 x 4096
[22:00] [DLC2SolstheimWorld] Creating texture atlas D:\Modding\tools\DynDOLOD\DynDOLOD_Output\textures\DynDOLOD\LOD\DynDOLOD_DLC2SolstheimWorld_Glow.dds 256 x 256
[22:55] <Error: Could not execute "D:\Modding\tools\DynDOLOD\Edit Scripts\Texconvx64.exe" -nologo -gpu 0 -y -m 1 -aw 256 -f R8G8B8A8_UNORM -srgb -o "C:\users\steamuser\Temp\DynDOLOD_SSE" "C:\users\steamuser\Temp\DynDOLOD_SSE\A6E3CDAB725A4765B65DCD2AC71CB4E5.dds">
[24:24] Error: A call to an OS function failed while processing textures\terrain\lodgen\dragonborn.esm\treepineforestash02_00017f73_2.dds.
[24:42] User says "Exit DynDOLOD"

I was not able to finish a complete Dyndolod run with all plugins enabled, but all of them finish when I select a single plugin after the other (like Tamriel or Sovngarde).

I have attached a link to the log and ini files on filebin. (The logs from second error I named to bugreport2 etc.)
https://filebin.net/qckm9psnivewi6r0



System Information: Proton 9 Stable / Ubuntu 24.04, Ryzen 5600G, 32 GB RAM, 32 GB Swap, RX 470 8 GB, > 90 GB free space, MO2 2.5.2.

[sheson]

6 minutes ago, ssuamier1 said:

I try to create dynamic LODs for Skyrim SE with DynDOLOD 3 Alpha 182 but I have random Errors with Texconv during generation.
Many of them are related to tree textures (from different mods). I have checked the textures but could not find any errors in them.

For example:

[01:26] <Error: Could not execute "D:\Modding\tools\DynDOLOD\Edit Scripts\Texconvx64.exe" -nologo -gpu 0 -y -m 1 -aw 256 -f R8G8B8A8_UNORM -srgb -o "C:\users\steamuser\Temp\DynDOLOD_SSE" "C:\users\steamuser\Temp\DynDOLOD_SSE\13E47B1E29F449EEAFAE3EA90369C1C3.dds">
[01:47] A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]
A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]
Error: A call to an OS function failed while processing HappyLittleTrees.esp TreePineForestSnowL03Dead [TREE:000EF59E]

I could recover from such errors via restart (and they do not reappear in the next run).
Later I got a kind of fatal error:

[21:58] [DLC2SolstheimWorld] Creating texture atlas D:\Modding\tools\DynDOLOD\DynDOLOD_Output\textures\DynDOLOD\LOD\DynDOLOD_DLC2SolstheimWorld.dds 8192 x 4096
[22:00] [DLC2SolstheimWorld] Creating texture atlas D:\Modding\tools\DynDOLOD\DynDOLOD_Output\textures\DynDOLOD\LOD\DynDOLOD_DLC2SolstheimWorld_Glow.dds 256 x 256
[22:55] <Error: Could not execute "D:\Modding\tools\DynDOLOD\Edit Scripts\Texconvx64.exe" -nologo -gpu 0 -y -m 1 -aw 256 -f R8G8B8A8_UNORM -srgb -o "C:\users\steamuser\Temp\DynDOLOD_SSE" "C:\users\steamuser\Temp\DynDOLOD_SSE\A6E3CDAB725A4765B65DCD2AC71CB4E5.dds">
[24:24] Error: A call to an OS function failed while processing textures\terrain\lodgen\dragonborn.esm\treepineforestash02_00017f73_2.dds.
[24:42] User says "Exit DynDOLOD"

I was not able to finish a complete Dyndolod run with all plugins enabled, but all of them finish when I select a single plugin after the other (like Tamriel or Sovngarde).

I have attached a link to the log and ini files on filebin. (The logs from second error I named to bugreport2 etc.)
https://filebin.net/qckm9psnivewi6r0

System Information: Proton 9 Stable / Ubuntu 24.04, Ryzen 5600G, 32 GB RAM, 32 GB Swap, RX 470 8 GB, > 90 GB free space, MO2 2.5.2.

This seems to be a duplicate post. See https://stepmodifications.org/forum/topic/19354-linuxproton-a-call-to-an-os-function-failed/page/4/#findComment-284145

https://dyndolod.info/Official-DynDOLOD-Support-Forum#Copy-and-Paste-Text
Do not post screenshots of the tools, unless it is about visual issues with the user interface. If necessary, convey the information in text form. The debug log as explained below typically contains most of the selected settings and options already.