I use Atmosphere 0.8.3 by fusee-primary.bin with Mupen64plus-next core (try original Mupen64 too), and always get this error when try lunch StarCraft 64
But on SXOS 2.5.3 - works, very slowly and with audio bugs
View attachment 158376
You held R with starting a game with SXOS right? To get the extra RAM.
If so it might just be that game is not fully playable.