The YouTube app uses webm 360p as maximum res but videos are scaled to "fit" 672x456, this means that videos are at incorrect aspect ratios. in both modes.
WiiMC does work well with h.264 on certain setups, encoding to 640x360(16:9) and 512x384(4:3) will always work, even with high compression. Or you can do what Netflix does, 512x384 with a 4:3 PAR will get you a higher quality 16:9 video, I've never tried this so I can't guarantee it would work in WiiMC, I have a 4:3 TV so I always go for 640x360.
The Netflix app generally looks really good but videos are always 512x384 max and unlike YouTube this one scales correctly.