I am having this same problem, and am curious if this is a AMD related issue? I have a 7950 X3D with 48 gigs of RAM and a Nvidia 4080 Super. The X3D series cores have a different way that they handle CPU cache.
Regardless, the set to single thread solution fixed the problem.