I built it on linux as well and ran into this same problem, with a nvidia 1650 super latest drivers from nvidia and Ubuntu 20.04. I moved the same binary to a system with intel integrated graphics and it runs perfect.
I suspect that it is some sort of issue with the nvidia GLX implementation...
I'm gonna buy another Series X, but can't decide if I should wait till the Xbox June event (maybe they get discounted) or just buy now. What do you guys think?