DSM that plays in The M3 media software generally isn't as good as the other two formats that Moonlights software uses, as far as i've heard.
I believe it's a custom variation of DPG, as the media player is based off moonshell orginally.
So if you want better video, you'll have to try out a homebrew application like moonshell or DSMplay.
Moonshell uses the DPG format which is similar to MPEG1. it gives reasonable quality at reasonable filesizes. (70ish megs for a 30 min vid)
DSMplay uses the DSM format (Not the same DSM as m3/g6 mov player uses). This is form of motion Jpeg (i think) and produces the best quality at the sacrifice of low filesizes (100s of megabytes for the same length video)
As for audio, neither Moonshell nor DSMplay sound muffled to me, they sound fine.