I was referring above all to the DSi XL, where GBA games appear excessively coarse, pixelated, angular, boxy
They're running at native resolution, just as they do on a DSi, DS lite, DS or GBA for that matter.
They look exactly as one should expect them to look on the DSi's screen. The "coarseness" you're talking about is just what the DSi XL's screen looks like. Due to its increased size it has a lower dot pitch than the others. This also affects DS games in exactly the same way and is just in the nature of the hardware.
To get any useful kind of smoothing you need a screen that's at least twice the resolution of the content you want to display.