If you think a bit technically, neither of them is better quality, as they both are container formats and no compression technology.
3GP uses the same specifications as MP4, which can contain stream data of any MPEG-4 compatible Video Codec (H.263, H.263+, XviD, DivX, H.264, MS MPEG 4 and other). The ability to play them on your device depends on the supported Codecs on it.
The same can be applied to DPG (abbreviation of nDsmPeG, "NDS MPEG"), which supports in DPG0 and DPG1 only MPEG1 compressed video and monoaural audio (stereo since DPG1), encoded with MP3. As of DPG3 you can also use XviD as Codec (requires MoonShell 2.00 or higher, supporting full screen with max. 25 FPS and wide-screen with max. 15 FPS). The DPG Container Format itself is a modified MPEG/AVI Container Format.
The quality question cannot be answered, as the quality itself depends on the source material and the re-encoding/compression process.